Thank you for this excellent article, which I almost passed by until I remembered what the original XOR problem was (so, for me at least, the title is inverted click-bait!)
With regard to the question of whether human thought is Turing complete: on the one hand, we clearly can solve Turing-hard problems up to the limits of our memory (a constraint which is considerably expanded through written language), while on the other hand, we can and do make mistakes while attempting such problems. Personally, I feel the former is sufficient to answer the question in the affirmative, despite of the fact of the latter.
The issue of Turing completeness is central to the Penrose-Lucas argument against AGI, as discussed at length in Roger Penrose's book "The Emperor's New Mind". Here, however, the claim is that human thought goes beyond Turing completeness: Gödel-unprovable results are provable by humans. While I feel that the complete argument against AGI fails on account of it equivocating over what it means to prove something, the above claim about the scope of human thought seems to be an empirical fact.
I have no idea how this observation could help in AI development, but we know that one can prompt a current-generation LLM to produce sentences that humans would recognize as expressing the sort of thoughts that would be useful in solving the given problem (though whether the LLM will go on to produce the solution that is obvious to the humans, and consistent with its earlier productions, is hit-or-miss.) This suggests to me that some sort of self-prompting capability might be necessary (but not sufficient) for a general problem-solving ability.
(...) Gödel-unprovable results are provable by humans.
Are they? Certainly humans can recognise the undedicability of a "barber's paradox", like "this sentence is false", but that doesn't mean we can actually decide its truth value. We can spot undecidability, but not resolve it.
I'm just being lazy and quoting a claim from Wikipedia's entry on the Penrose-Lucas argument, allegedly referencing "The Emperor's New Mind", but without any specific location within that book. It has been a while since I read it, and I may attempt to find a more precise summary of Penrose's claim (he has written about it extensively elsewhere, and so have many other people) but I can't do that immediately.
I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption.
What seems like an even better formal solution is something like logical induction[0] that works over something like probabilities instead of truth/falsehood. It has an existence proof but not a constructive formalization, afaik, but maybe some recent progress has been made.
Thanks for the paper, I need to read that. Much of my work is on inductive logic.
>> I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption.
What you describe sounds to me more like logical abduction, rather than induction. Unfortunately, it's difficult to find simple explanations of what abduction is. One very rough and operational categorisation of modes of inference, as deductive, inductive or abductive, is as follows. Deduction is reasoning (as in automated theorem proving or rule-based systems), induction is learning (as in machine learning) and abduction is reasoning under uncertainty (as in probabilistic inference).
There's a lot of confusion between what is abduction and what is induction, and the definitions of those in terms of "facts" and "rules" (coming from Peirce, who originally named "abduction" at least) is, for me, not very helpful. Another colloquial explanation is that abduction is a form of drawing conclusions from observations that isn't sound, so that conclusions are not necessarily true. I think that's where the association with probabilistic reasoning comes from, and that's why I think it describes better the kind of induction the linked paper is talking about.
Thanks for clarifying! Sorry for pouncing on that one little detail in your comment btw. Incompleteness vs. human reasoning is just something I think about on and off and your turn of phrase caught my attention.
Update: One of the things I overlooked when using that quote is that it appears to say that we can prove the truth of any undecidable sentence, but Penrose's argument is made only with respect to the formalization, in a consistent and sufficiently strong [1] system S, of the sentence "the truth of this sentence is not provable in S". Both Lucas and Penrose claim that we will see that it is true (but, as it asserts, not provable in S) as the alternative leads to contradiction.
This depends on S being consistent. Lucas and Penrose apparently suppose that any proposal for explaining thinking as a computational process would, ipso facto, cast a mind as a concrete instantiation of a consistent formal system. [2]
Putting aside any thoughts of whether this is persuasive, I think this seems to be a case of Lucas and Penrose doing what they claim is uniquely human: thinking outside of the box of formal proof. More generally, almost none of human knowledge is couched as a formal proof of something or other, and almost none of it has been acquired that way.
[1] More precisely, one in which a “moderate amount of number theory” can be proven.
>> Putting aside any thoughts of whether this is persuasive, I think this seems to be a case of Lucas and Penrose doing what they claim is uniquely human: thinking outside of the box of formal proof. More generally, almost none of human knowledge is couched as a formal proof of something or other, and almost none of it has been acquired that way.
Well, except for all of mathematics :P
Then again, it's useful to wonder what, exactly, is a formal proof, when derived by a human. My understanding of proofs (after having had to come up with several for my PhD research) is that the purpose of a proof is to convince a human that what the associated theorem states, is true. The human is first of all the person stating the theorem, so for example these days at least when I write down a theorem, the next thing I do is ask myself out loud "why do I think this is true?". Then I try to write it down in simple English, and then I try to put it in the standard format of an inductive proof, or a proof by refutation- and I do that only because those are proof structures that are well understood and they will cause the least confusion about what I mean. Even so I've got proofs wrong and I had reviewers pick out my mistakes and help me correct them, which is a bit of a "phew, dodged a bullet" (worst case is you get your erroneous proof published and look like an idiot for eternity).
Btw, I picked up that idea that the purpose of proofs is to convince others (other mathematicians) from some article or paper I read about proofs in mathematics, but I have no idea which one it was now. Anyway it's not my own idea, I just found it somewhere and it made a lot of sense, and still does.
Then there's the proofs you can construct with an automated theorem prover. I wonder if that's more the kind of proof that Penrose is talking about? In a sense, I don't think any human can really really construct an entire proof. All the proofs I've seen (and all the ones I've written) are more correctly proof sketches. That's why I think their point is only to convince others that what you say is true, and not to uncover some fundamental truth about the nature of the universe. Humans don't do that. I don't think we can do that. Formal systems like arithmetic or logic try to capture what is our ideal reasoning procedure, but not what we do in practice. I think so anyway.
>> This depends on S being consistent. Lucas and Penrose apparently suppose that any proposal for explaining thinking as a computational process would, ipso facto, cast a mind as a concrete instantiation of a consistent formal system.
I haven't read Lucas' and Penrose's work but I would find it very hard to be convinced about the truth of that.
I don't understand how humans reason. I can see what I do, and what others do, but after a certain point all models seem to break down and minds seem capable of just jumping to conclusions because they like them. It's such a huge question how our minds work, exactly. I will probably never know.
I see your point about mathematics, but even if most of mathematics were done by formally proving theorems, I would argue that it is still a small (though highly significant) part of the whole of human knowledge, when we take into account everything else studied in academe beyond mathematics and logic, together with the knowledge that each person uses in their everyday lives. I am not a mathematician and I struggle with it, but I have read that most mathematics is not conducted formally, but with informal reasoning which mathematicians reasonably expect could be formalized, if it came to that - a view that seems closely aligned with what you have written here (I'm not sure where I read this; possibly a Quanta article by someone such as Terence Tao.) If my history is correct, a lot of the mathematics that we use outside of mathematics itself was developed before it was formalized.
Don't put too much trust in what I wrote about how Penrose deals with the consistency issue; it's been a long time since I read his first book, I never did read the second (his response to criticism of the first), and I put that paragraph together from a couple of hours 'refresher' on the internet. I am reasonably sure, however, in my recollection that Penrose, following Lucas, constructed his argument from the formality of Turing machines and the Turing-equivalence of programming languages. Theorem provers demonstrate that it is possible for computers to produce formal arguments, but that is not sufficient for his purpose, which is to show that computers cannot do something that human minds can do.
On the other hand, I am guessing that it would be relatively straightforward to take a theorem-prover and modify it so that its logic is inconsistent? If I am not mistaken, this is a question you are well-positioned to answer, so your comments would be appreciated. If inconsistent theorem "provers" are possible, this would seem to defeat Penrose's argument, unless one adds the premise that humans are consistent reasoners, which seems to me to be obviously false.
Personally, I believe that human minds have evolved via natural selection from precursors that are, broadly speaking, understandable, and have probably done so without the intercession of currently-unknown fundamental physics or extra-physical causes, so I take an interest in arguments claiming that this cannot possibly be so, or that they could not possibly be simulated by a digital computer. Nevertheless, I agree entirely with your last paragraph.
>> On the other hand, I am guessing that it would be relatively straightforward to take a theorem-prover and modify it so that its logic is inconsistent? If I am not mistaken, this is a question you are well-positioned to answer, so your comments would be appreciated. If inconsistent theorem "provers" are possible, this would seem to defeat Penrose's argument, unless one adds the premise that humans are consistent reasoners, which seems to me to be obviously false.
Yes, you could modify an automated theorem prover to be inconsistent, or unsound, in fact you could do that in many different ways. My specialty is Resolution-based theorem proving. While Resolution is sound and complete in principle, in practice it doesn't make sense to consider it so except under the Closed World Assumption (CWA). The CWA, very simply stated is that only what is currently known, either as axiomatic truths or theorems derived from those, is true, and everything else is false. That might sound dumb, but it's actually one of the simplest ways to deal with uncertainty: anything that can't be known with certainty is set aside. The good thing of course is that the CWA does not have to be seen as a static assumption. In particular, it is a very well known fact that if new information is received, under a CWA (more precisely, with Negation-As-Failure, under a CWA) the set of logical consequences of a theory can both increase, and decrease - non-monotonically. What that means, in practice, is that, under the CWA, beliefs can be updated as new information comes in, so that a reasoner doesn't have to be stuck forever in a "bad world" entailed by a faulty set of initial assumptions.
In a sense, given a sound and complete inference rule like Resolution, the CWA means a reasoning system can be "locally consistent" without having to be "globally consistent". It can be right for what it knows for sure, without having to worry too much about what it doesn't know, because once unknowns become known, local consistency kicks in and the reasoner's model of the world is updated to better approximate reality. Unfortunately most of that remains mainly theoretical. To my knowledge nobody has really gone and built a system that learns continuously by exploiting the non-monotonicity of Resolution under a CWA. Although people have build systems, they tend to be academic prototypes, that are only described in a couple of papers at most and then forgotten about.
There's another way in which Resolution can be inconsistent ("inconsistent" is a very loose term, I hope I'm using it right :). That is by performing induction, or abduction. Normally, Resolution is deductive: one has a theory (in practice that's a logic program, a set of definite clauses) and proves new consequences of the theory. In the Inductive use of Resolution ("Meta-Interpretive Learning"... doesn't quite roll off the tongue, I know) one has some background theory and some observations not covered by the theory, and Resolution is used to add clauses to the theory so that it "covers" the new observations. In abduction, the premises of the theory, its very axioms, are updated.
In both cases, reasoning is unsound. In induction, because we can never observe every logical consequence of a new theory that we derive from some small number of observations, so we can never verify the theory (but we can falsify it). In abduction, because axioms are by their nature unsound, arbitrary assumptions that are not provable and we only hold true by choice.
I guess all this is very abstract and will sound vague, so sorry about that. Note also that it's all a drastic departure from classical logic, that assumes that everything to be known is known, and knowable (although maybe undecidable). There's a certain opinion that non-monotonic reasoning is closer to human reasoning, because of its ability to be locally consistent and globally who-cares. I don't know about human-like, but it certainly seems like a good way to approach the real world, with all its uncertainties, where we nevertheless want to have good answers that we can rely on.
Let me know if you would like to read further on all this and I can give some pointers.
Thanks for your reply. I agree that non-monotonic logic seems a more useful way to deal with an uncertain real world than classical logic. I would be interested in following up on a few pointers, though I don't want you to spend too much effort on it, as I can't guarantee I have either the time or the aptitude to take it too far.
On the memory limitation: Yeah, lack of an infinite tape is a real bummer. On some level I think we're some kind of bounded-tape automaton, which still makes the current limitations of Transformers pretty severe. (Yes, yes, prompting, CoT, and all that)
With regard to the question of whether human thought is Turing complete: on the one hand, we clearly can solve Turing-hard problems up to the limits of our memory (a constraint which is considerably expanded through written language), while on the other hand, we can and do make mistakes while attempting such problems. Personally, I feel the former is sufficient to answer the question in the affirmative, despite of the fact of the latter.
The issue of Turing completeness is central to the Penrose-Lucas argument against AGI, as discussed at length in Roger Penrose's book "The Emperor's New Mind". Here, however, the claim is that human thought goes beyond Turing completeness: Gödel-unprovable results are provable by humans. While I feel that the complete argument against AGI fails on account of it equivocating over what it means to prove something, the above claim about the scope of human thought seems to be an empirical fact.
I have no idea how this observation could help in AI development, but we know that one can prompt a current-generation LLM to produce sentences that humans would recognize as expressing the sort of thoughts that would be useful in solving the given problem (though whether the LLM will go on to produce the solution that is obvious to the humans, and consistent with its earlier productions, is hit-or-miss.) This suggests to me that some sort of self-prompting capability might be necessary (but not sufficient) for a general problem-solving ability.