New tool maps the landscape of student knowledge using short quizzes
A Dartmouth study reports a mathematical framework that could power next-gen AI systems to enhance personalized learning.
When we learn something new, that information does not exist in isolation. It integrates into the complex landscape of our knowledge, forging connections with existing ideas and opening up possibilities for new learning.
In a new study in Nature Communications, Dartmouth researchers report a mathematical technique for mapping the unique landscape of a student's conceptual knowledge from their performance on short multiple-choice quizzes. Their framework turns a traditional quiz into a detailed topography that captures the peaks of a student's conceptual mastery and the valleys where they struggle.
According to the researchers, this knowledge-mapping technique could be used to enhance classroom learning by providing educators with a way to automatically identify the concepts individual students do and do not understand, track how their understanding evolves as they learn, and determine how to best connect new concepts to their existing knowledge.
Outside of the classroom, the framework could power a new generation of personalized AI tutors capable of deeply understanding students' knowledge and tailoring their feedback accordingly.
The framework seeks to address a fundamental limitation of traditional learning assessments, says Jeremy Manning, the study's senior author and an associate professor of psychological and brain sciences at Dartmouth.
"When a student scores 50% on a quiz, that number conveys little about what they actually understand," Manning says. "They may have understood half of the material perfectly, or understood all of it only partially, or anywhere in-between."
"Our approach leverages the intuition that people's knowledge tends to vary gradually across related ideas—that knowing a lot about one concept suggests you're more likely, though not guaranteed, to also know something about related concepts."
Paxton Fitzpatrick, lead author of the study and a PhD candidate in Manning's research group, says the framework is a step toward creating AI tutoring systems that can adapt to the needs of individual learners no matter where they are.
"When a student seeks help after struggling on an exam, providing them with individualized feedback or guidance requires examining their performance on different questions to better understand what concepts they have and haven't mastered," Fitzpatrick says.
"While that's traditionally the role of a teacher or tutor, the growth of online and remote learning means that sort of personalized instruction isn't always available to every student," he says.
Concepts as coordinates
To characterize how different concepts relate to each other, the researchers used text embedding models—the same class of models powering modern AI systems—to represent concepts as coordinates in a high-dimensional space.
In this space, conceptual similarity is captured by physical distance: related concepts such as gravity and magnetism are located near each other while unrelated concepts like genetics and art history remain distant. By assigning coordinates to each quiz question a student answered, the researchers are able to infer their level of knowledge about other concepts that live nearby.
"Our goal wasn't just to build better quizzes or grading methods," Manning says. "We wanted to test a theoretical idea, that knowledge is structured in the particular way implied by text embedding models, and that this structure shapes what people are likely to know and how they acquire new knowledge.
"If this is true, we can leverage that to characterize what someone knows much more quickly and efficiently than just considering one concept or question at a time," he says.
In their study, the researchers mapped 50 Dartmouth undergraduates' knowledge before and after they viewed online lectures from Khan Academy, a popular nonprofit educational company. They report in Nature Communications that knowledge maps for individual students not only captured changes in their knowledge as they learned from the lectures, but reliably predicted which quiz questions they'd be able to correctly answer.
Mirroring top teachers
Study co-author Andrew Heusser, who worked on the project as a postdoctoral researcher in Manning's lab, says the mathematical process underlying this tool mirrors the mental process teachers and tutors use to decide how to reframe or clarify concepts a student struggled with on an exam. They relate new ideas to concepts the student already knows.
"When discussing a new concept with an individual student, a teacher will leverage a sort of 'mental map' of that student's knowledge to inform how they might frame that concept or relate it to other ideas they know that student is already familiar with," Heusser says. "Our framework is essentially an attempt to mathematically approximate that abstract mental map."
The team also is careful to note that AI tutoring systems should not be seen as a replacement for teachers. "In small classrooms or with one-on-one tutoring, human teachers can understand their students far better than any Al can," Fitzpatrick says.
"The challenge we're forced to reckon with is that highly personalized education can't scale to all of humanity," he says. "We think the future is in these new tools that can help broaden the reach of educators by supplementing some of the aspects that make the best teachers so effective."
The researchers released a public demo of their framework where users can answer questions to construct an interactive map of their knowledge, view their predicted areas of expertise, and engage with recommended educational materials to fill gaps in their understanding or expand their knowledge.
END
In a new study in Nature Communications, Dartmouth researchers report a mathematical technique for mapping the unique landscape of a student's conceptual knowledge from their performance on short multiple-choice quizzes. Their framework turns a traditional quiz into a detailed topography that captures the peaks of a student's conceptual mastery and the valleys where they struggle.
According to the researchers, this knowledge-mapping technique could be used to enhance classroom learning by providing educators with a way to automatically identify the concepts individual students do and do not understand, track how their understanding evolves as they learn, and determine how to best connect new concepts to their existing knowledge.
Outside of the classroom, the framework could power a new generation of personalized AI tutors capable of deeply understanding students' knowledge and tailoring their feedback accordingly.
The framework seeks to address a fundamental limitation of traditional learning assessments, says Jeremy Manning, the study's senior author and an associate professor of psychological and brain sciences at Dartmouth.
"When a student scores 50% on a quiz, that number conveys little about what they actually understand," Manning says. "They may have understood half of the material perfectly, or understood all of it only partially, or anywhere in-between."
"Our approach leverages the intuition that people's knowledge tends to vary gradually across related ideas—that knowing a lot about one concept suggests you're more likely, though not guaranteed, to also know something about related concepts."
Paxton Fitzpatrick, lead author of the study and a PhD candidate in Manning's research group, says the framework is a step toward creating AI tutoring systems that can adapt to the needs of individual learners no matter where they are.
"When a student seeks help after struggling on an exam, providing them with individualized feedback or guidance requires examining their performance on different questions to better understand what concepts they have and haven't mastered," Fitzpatrick says.
"While that's traditionally the role of a teacher or tutor, the growth of online and remote learning means that sort of personalized instruction isn't always available to every student," he says.
Concepts as coordinates
To characterize how different concepts relate to each other, the researchers used text embedding models—the same class of models powering modern AI systems—to represent concepts as coordinates in a high-dimensional space.
In this space, conceptual similarity is captured by physical distance: related concepts such as gravity and magnetism are located near each other while unrelated concepts like genetics and art history remain distant. By assigning coordinates to each quiz question a student answered, the researchers are able to infer their level of knowledge about other concepts that live nearby.
"Our goal wasn't just to build better quizzes or grading methods," Manning says. "We wanted to test a theoretical idea, that knowledge is structured in the particular way implied by text embedding models, and that this structure shapes what people are likely to know and how they acquire new knowledge.
"If this is true, we can leverage that to characterize what someone knows much more quickly and efficiently than just considering one concept or question at a time," he says.
In their study, the researchers mapped 50 Dartmouth undergraduates' knowledge before and after they viewed online lectures from Khan Academy, a popular nonprofit educational company. They report in Nature Communications that knowledge maps for individual students not only captured changes in their knowledge as they learned from the lectures, but reliably predicted which quiz questions they'd be able to correctly answer.
Mirroring top teachers
Study co-author Andrew Heusser, who worked on the project as a postdoctoral researcher in Manning's lab, says the mathematical process underlying this tool mirrors the mental process teachers and tutors use to decide how to reframe or clarify concepts a student struggled with on an exam. They relate new ideas to concepts the student already knows.
"When discussing a new concept with an individual student, a teacher will leverage a sort of 'mental map' of that student's knowledge to inform how they might frame that concept or relate it to other ideas they know that student is already familiar with," Heusser says. "Our framework is essentially an attempt to mathematically approximate that abstract mental map."
The team also is careful to note that AI tutoring systems should not be seen as a replacement for teachers. "In small classrooms or with one-on-one tutoring, human teachers can understand their students far better than any Al can," Fitzpatrick says.
"The challenge we're forced to reckon with is that highly personalized education can't scale to all of humanity," he says. "We think the future is in these new tools that can help broaden the reach of educators by supplementing some of the aspects that make the best teachers so effective."
The researchers released a public demo of their framework where users can answer questions to construct an interactive map of their knowledge, view their predicted areas of expertise, and engage with recommended educational materials to fill gaps in their understanding or expand their knowledge.
END