On Sun, 26 Mar 2023 at 19:07, William Stein <wst...@gmail.com> wrote: > > Ok, that "proof" from GPT-4 is pretty absurd nonsense to put it mildly. I > wonder if there will be a new surge in crank math papers this year.
Apparently the way that the chat GPT models are trained is that the source data comes from having humans use them and then rate the quality of the answers they get given. Then they train a model to predict how humans would rate the answers and then they use that to train new iterations of GPT. After some iterations of that they go back to the humans again and so on. What that means is that ultimately the target of the chat models is to try to satisfy the humans who are using them in testing. They have used any old humans though rather than say "experts" so the goal is not to be "correct" but just to try to satisfy the human users. One implication of this training goal is that the models are optimised towards giving superficially plausible answers. A clever sounding but incorrect answer has a chance to satisfy a human who does not read carefully. A negative result like "Sorry I can't answer" is likely to receive a poor rating from most humans even if it is the most correct answer. Also these models will actually try to judge what sort of human you are. If your question suggests that you do know what you are talking about then the bot will try to give an answer that would please someone who knows what they are talking about. Naturally the converse applies as well. This means that the wording of your question can alter the answers that you receive in more ways than you might immediately expect. These models are called language models for good reason because they are really just trained to be good at language. Their ability to answer questions that seem to involve some reasoning is no different from a human BS-monger who can google for a bit and knows how to string some sentences together in a way that momentarily resembles the language of someone who knows what they are talking about. The limits of their actual reasoning are quite clear in the final part of this proof where we go from a theorem like algebraic^algebraic -> transcendental in one step to transcendental^transcendental -> transcendental. Quite apart from the bogus algebra this is a failure in pretty elementary logic. However I think that Chat GPT on some level *knows* that the logic is bogus. It has just scored that bogusness and decided that it is better than the alternatives it could generate for the problem at hand (and the user at hand!). If you castigate the bot and point out its fallacies or even just tell it lies then it will rework its answer to be some new BS it thinks you will be more likely to be satisfied by. -- Oscar -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxRw-b1KPe9DwwgPgK1-H9K36FBXYzT7L1PNBr11qerf%3Dw%40mail.gmail.com.
