US students and science: AAAS testing gives new insight on what they know and their misconceptions

(Press-News.org) The American Association for the Advancement of Science has launched an innovative website to help educators assess more precisely what students know about key ideas in science and -- just as importantly -- the incorrect ideas they have. It offers an unusually detailed picture of how middle and high school students across the United States are currently doing in science, and features information on what they know and on hundreds of misconceptions they have about everything from the size of atoms to whether all organisms have DNA.

Knowing these misconceptions and how pervasive they are -- which is not typically part of the analysis of test results from state testing or from leading national and international testing organizations -- can help teachers improve instruction and better design their own test questions to assess whether students truly understand the science concepts they are being taught.

The newly developed test questions also counter the widely held view that multiple choice questions are useful only for testing recall of memorized definitions and trivial facts, says George DeBoer, deputy director of Project 2061, the AAAS program that developed the new assessment website.

"As a result of our efforts, many of the test questions included in the new website measure not only knowledge of factual information, but they also probe a student's ability to explain real-world phenomena, reason logically through problem situations, or identify the reason why a claim is true," DeBoer said.

Multiple-choice testing has some clear advantages, he said. It tends to be less costly and easier to administer and score than other types of testing. "It is unlikely to disappear from the assessment landscape," he said. "The goal of the project, then, was to improve on the design and use of multiple-choice questions."

But the focus on multiple-choice testing should not be misinterpreted as an endorsement of it as the best approach to student assessment, DeBoer cautioned. "Any type of assessment can be used effectively," he added, "as long as attention is paid to certain fundamental design principles."

The new AAAS website (http://assessment.aaas.org) presents detailed information on how a national sample of students answered each question, along with an analysis of both their correct and incorrect responses. In contrast, national and international assessment programs such as the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program in International Student Assessment (PISA) report on the percent of students in participating states or countries who answer items correctly but do not report on the incorrect ideas students have, DeBoer said.

Project 2061, founded in 1985 by AAAS to improve science education, developed the assessment items and collected data on them under a grant from the National Science Foundation. The website, where the results of this work can be found, includes data on student performance on the test questions by gender, grade level, and whether or not English is the student's primary language. Each question typically was answered by at least 2000 students in field tests involving school districts across the nation. In 2010, for example, more than 90,000 students in 814 schools participated in the field tests. Project 2061 researchers also conducted on-site interviews with students to gauge the effectiveness of the questions.

Constructing good multiple-choice questions requires knowledge of science and of effective assessment practices. "Too often," DeBoer said, "test questions are not linked explicitly to the ideas and skills that the students are expected to learn." And too often the questions are written in a way that makes it difficult for students to understand what is being asked, he said.

With increasing calls for national science standards and a common core curriculum for middle-school and high-school students, it is more important than ever to get assessment right, DeBoer said. "Good assessments can be used to actually improve students' learning and not just to hold teachers and schools accountable," he added. "Assessments that are designed to diagnose students' misconceptions can be powerful educational tools."

For the key idea that "all matter is made up of atoms," the AAAS website notes that 27% of the middle school students and 20% of the high school students who were tested incorrectly believed that "cells are not made up of atoms." On the key idea that "genetic information is encoded in DNA molecules," 40% of middle school students and 30% of high school students had the misconception that only animals have DNA while plants and mushrooms do not.

Having information about such misconceptions is just as important as knowing how many students answered a question correctly, DeBoer said. It can reveal gaps that prevent a coherent understanding of the topic.

The AAAS website is organized to quickly give users a picture of what students know and the misconceptions they have. The test questions for a specific science concept are listed together, arranged from the highest to lowest percent correct. The misconceptions also are listed in order, with the misconceptions selected most frequently at the top of the list.

So, for example, there are 20 questions probing the idea that "genetic information is encoded in DNA molecules." At the top of the correct responses, 94% of middle school students knew that "DNA is the molecule that contains genetic information that is passed from parents to offspring." At the bottom of the list, only 15% of the students knew that "Genes are chemical sequences of nucleotides, not the actual expression of characteristics or traits."

"This is extremely valuable information for teachers and curriculum developers to have" said DeBoer, "because it shows them where instruction needs to be targeted."

The website has received early positive reviews from teachers and curriculum specialists who attended a recent workshop at AAAS on the methods used to develop the test items.

The emphasis on student misconceptions is particularly helpful, said Deagan Andrews, a curriculum and assessment specialist in the Greeley, Colorado, school district. Referring to some of the major standardized test programs, he said: "No one releases any information about misconceptions. They are interested in whether students got it right or wrong." He said the emphasis on misconceptions "is a critical piece that has been missing," offering teachers an opportunity to "think about their instruction and what they may be doing that may be perpetuating misconceptions."

"Students create strange conceptions about the world from their experiences," agreed Anu Malipatil, a school administrator for a network of charter schools in New York and Connecticut. "It becomes more difficult to teach students without actually addressing the misconception first." She said most state tests do not provide an analysis of incorrect answers. The AAAS test items, she said, can be used by teachers to diagnose potential problem areas and find out how many students in a classroom have misconceptions about key concepts.

Teresa Eastburn, a former classroom science teacher who is now an education and outreach specialist at the National Center for Atmospheric Research in Boulder, Colorado, said the assessment site ultimately benefits teachers as well as students, particularly teachers who may not have an extensive background in science. Because of the level of detail, including the emphasis on misconceptions, the site "will really inform teaching," Eastburn said.

### END