They have no ability to actually reason

I'm curious about this kind of statement. "Reasoning" is not a clearly defined scientific term, in that it has a myriad different meanings depending on context.

For example, there has been science showing that LLMs cannot use "formal reasoning", which is a branch of mathematics dedicated to proving theorems. However, the majority of humans can't use formal reasoning. This would make humans "unable to actually reason" and therefore not Generally Intelligent.

At the other end of the spectrum, if you take a more casual definition of reasoning, for example Aristotle's discursive reasoning, then that's an ability LLMs definitely have. They can produce sequential movements of thought, where one proposition leads logically to another, such as answering the classic : "if humans are mortal, and Socrates is a human, is Socrates mortal ?". They demonstrate the ability to do it beyond their training data, meaning they do encode in their weights a "world model" which they use to solve new problems absent from their training data.

Whether or not this is categorically the same as human reasoning is immaterial in this discussion. The distinct quality of human thought is a metaphysical concept which cannot be proved or disproved using the scientific method.