Random thoughts
I may contradict myself by the end or maybe there was no point to make.
There are plethora of fields where AI is making progress or at least, envisoned to solve problems with the technology getting better. People are good in cutting the search space by a billion-fold. People with AI can multiply the effect. But is that true?
It is possible to reduce search space trillion-fold with having the AI accelerate the search by going and verifying potential paths. But for a moment, lets view the AI in isolation. They are next token preditors. Language models produe the next word, not just any word, but a specific word out of the all the possible words in English dictionary (including producing no word at all). Predicting the next word is a way of cutting the search-space. When LLMs produce the next token, what is to going to come after and till the end is already specified in its distribution. The distribution changes every time the model produces a token. The parameters such as greedy, top-p, temperature, etc. just change how you draw from the distribution, but it can't be outside the distribution. Large Language Model doesn't know what it knows and doesn't know what it doesn't know.
Its obvious to see that prediction in LLMs are different from humans. Human predition is more complex that the next word. Humans can make predicitons of longer timescales, not just what the next sentence will be. Whether the prediciton holds is a different matter (long term predictions tend to be less reliable in general). This makes a different kind of search problem. Humans can cut the search space with better understanding of the problem and sniping on important events not autoregressively.
Some people will point to scientific results from fields outside their own that are marketed as having been done with "AI" and ask: How could you take a hard line against "AI" when it has provided XYZ? Such remarks both conflate many different things under the umbrella of “AI” and are also usually based on hype-filled media coverage that tends to obscure what technology was actually used in what way. My hypothesis on use of AI in open ended research is that its not fulfilling. Current AI can't come up with better explanations or thoughful experiments so one would use them to explore ideas where a rival theory already exists. It can still be a powerful tool for generating conjecturs or speeding up the boring parts so humans can iterate faster.