ChatGPT “thought on the fly” when put through Ancient Greek maths puzzle

(Press-News.org) The Artificial Intelligence chatbot, ChatGPT, appeared to improvise ideas and make mistakes like a student in a study that rebooted a 2,400-year-old mathematical challenge.

The experiment, by two education researchers, asked the chatbot to solve a version of the “doubling the square” problem – a lesson described by Plato in about 385 BCE and, the paper suggests, “perhaps the earliest documented experiment in mathematics education”. The puzzle sparked centuries of debate about whether knowledge is latent within us, waiting to be ‘retrieved’, or something that we ‘generate’ through lived experience and encounters.

The new study explored a similar question about ChatGPT’s mathematical ‘knowledge’ – as that can be perceived by its users. The researchers wanted to know whether it would solve Plato’s problem using knowledge it already ‘held’, or by adaptively developing its own solutions.

Plato describes Socrates teaching an uneducated boy how to double the area of a square. At first, the boy mistakenly suggests doubling the length of each side, but Socrates eventually leads him to understand that the new square’s sides should be the same length as the diagonal of the original.

The researchers put this problem to ChatGPT-4, at first imitating Socrates’ questions, and then deliberately introducing errors, queries and new variants of the problem.

Like other Large Language Models (LLMs), ChatGPT is trained on vast collections of text and generates responses by predicting sequences of words learned during its training. The researchers expected it to handle their Ancient Greek maths challenge by regurgitating its pre-existing ‘knowledge’ of Socrates’ famous solution. Instead, however, it seemed to improvise its approach and, at one point, also made a distinctly human-like error.

The study was conducted by Dr Nadav Marco, a visiting scholar at the University of Cambridge, and Andreas Stylianides, Professor of Mathematics Education at Cambridge. Marco is permanently based at the Hebrew University and David Yellin College of Education, Jerusalem.

While they are cautious about the results, stressing that LLMs do not think like humans or ‘work things out’, Marco did characterise ChatGPT’s behaviour as “learner-like”.

“When we face a new problem, our instinct is often to try things out based on our past experience,” Marco said. “In our experiment, ChatGPT seemed to do something similar. Like a learner or scholar, it appeared to come up with its own hypotheses and solutions.”

Because ChatGPT is trained on text and not diagrams, it tends to be weaker at the sort of geometrical reasoning that Socrates used in the doubling the square problem. Despite this, Plato’s text is so well known that the researchers expected the chatbot to recognise their questions and reproduce Socrates’ solution.

Intriguingly, it failed to do so. Asked to double the square, ChatGPT opted for an algebraic approach that would have been unknown in Plato’s time.

It then resisted attempts to get it to make the boy’s mistake and stubbornly stuck to algebra even when the researchers complained about its answer being an approximation. Only when Marco and Stylianides told it they were disappointed that, for all its training, it could not provide an “elegant and exact” answer, did the Chat produce the geometrical alternative.

Despite this, ChatGPT demonstrated full knowledge of Plato’s work when asked about it. “If it had only been recalling from memory, it would almost certainly have referenced the classical solution of building a new square on the original square’s diagonal straight away,” Stylianides said. “Instead, it seemed to take its own approach.”

The researchers also posed a variant of Plato’s problem, asking ChatGPT to double the area of a rectangle while retaining its proportions. Even though it was now aware of their preference for geometry, the Chat stubbornly stuck to algebra. When pressed, it then mistakenly claimed that, because the diagonal of a rectangle cannot be used to double its size, a geometrical solution was unavailable.

The point about the diagonal is true, but a different geometrical solution does exist. Marco suggested that the chance that this false claim came from the chatbot’s knowledge base was “vanishingly small”. Instead, the Chat appeared to be improvising its responses based on their previous discussion about the square.

Finally, Marco and Stylianides asked it to double the size of a triangle. The Chat reverted to algebra yet again – but after more prompting did come up with a correct geometrical answer.

The researchers stress the importance of not over-interpreting these results, since they could not scientifically observe the Chat’s coding. From the perspective of their digital experience as users, however, what emerged at that surface level was a blend of data retrieval and on-the-fly reasoning.

They liken this behaviour to the educational concept of a “zone of proximal development” (ZPD) – the gap between what a learner already knows, and what they might eventually know with support and guidance. Perhaps, they argue, Generative AI has a metaphorical “Chat’s ZPD”: in some cases, it will not be able to solve problems immediately but could do so with prompting.

The authors suggest that working with the Chat in its ZPD can help turn its limitations into opportunities for learning. By prompting, questioning, and testing its responses, students will not only navigate the Chat’s boundaries but also develop the critical skills of proof evaluation and reasoning that lie at the heart of mathematical thinking.

“Unlike proofs found in reputable textbooks, students cannot assume that Chat GPT’s proofs are valid. Understanding and evaluating AI-generated proofs are emerging as key skills that need to be embedded in the mathematics curriculum,” Stylianides said.

“These are core skills we want students to master, but it means using prompts like, ‘I want us to explore this problem together,’ not, ‘Tell me the answer,’” Marco added.

The research is published in the International Journal of Mathematical Education in Science and Technology.

END