In a recent interview on this blog, Dr. Pooja K. Agarwal spoke about the benefits of retrieval practice: a study technique that–in her words–focuses on pulling information OUT of students’ brains rather than getting it back IN.
For example: if I begin today’s class by having my students write down three things they remember from yesterday’s lesson on the Han dynasty, that’s retrieval practice. After all, they’re going back into their memories and drawing OUT facts and ideas we discussed.
If, however, I begin by briefly summarizing yesterday’s class, well, then I’m trying to put information back IN. That’s not retrieval practice.
Dr. Agarwal summarizes the benefits of retrieval practice thus: “it works for all students in all subjects, all the time.”
Sounds tempting, no?
Pushing Boundaries
In one part of our conversation, Dr. Aragwal notes that she likes doing research in actual classrooms with actual students–rather than in psychology labs in highly controlled conditions–because “I really like the messiness of of doing scientific research in classrooms. The fire alarms, and school assemblies, and kids who are out sick, I really enjoy it because it pushes boundaries.”
In the spirit of messiness, here’s a recent post from the Learning Scientists about using retrieval practice in elementary school to learn vocabulary.
The good news about this study:
First: it took place in a real school with real students, not in a psychology lab. That means its results are likelier to be meaningful to teachers.
Second: the participants were 9-year-olds, not college students. So, we can be more confident that retrieval practice works with…say…4th graders.
Third: the study took place in the Netherlands, so we’ve got reason to believe that the benefits go beyond a North American cultural context.
So far, so good.
Let the Messiness Begin
At the same time, this particular study revealed a few muddles as well.
Muddle #1: the size of the benefit was relatively small. Retrieval practice produced more learning than simple restudy, and more than “elaborative retrieval,” but statistically speaking that difference was harder to find than in a psychology lab.
Muddle #2: Dr. Agarwal’s research shows that fill-in-the-blank retrieval practice and multiple-choice retrieval practice are equally effective. This study, however, contradicts that finding; multiple-choice retrieval didn’t produce more learning than pure restudy.
Muddle #3: believe it or not, muddle #3 contradicts muddle #2. Because of the study design, the authors acknowledge that their own findings about multiple-choice tests aren’t fully persuasive. For example: because the average score on the multiple-choice tests was above a 90%, there wasn’t enough difference among the students’ scores to calculate meaningful effects.
What should teachers do with all this contradictory information?
My advice: Embrace the muddle.
Teachers should expect that different studies produce muddled–and occasionally contradictory–results.
No one study tells us everything we need to know about retrieval practice. Instead, we’re looking for patterns of findings.
If we do ten studies, and eight of them show that retrieval practice helps learning, that’s impressive. We don’t need to be thrown off by one study that shows no effect–or, as in this case, a relatively smaller effect than in a psych lab.
The Quiet Finding
Although the authors don’t dwell on this point, one finding jumped out at me.
In one of the restudy conditions, students were asked to “elaborate” on the meaning of the word. For example, as they tried to remember “compost pile,” they were asked to circle the words relating to a compost pile on this list: manure, plastic, delicious, orange-peels, mailbox, dead leaves.
My teacherly instincts tell me that this restudy condition ought to help students. After all, to circle the correct words, they have to think a bit harder about the meaning of the phrase “compost pile.” That additional thought strikes me as a desirable difficulty, and ought to produce more learning.
But–at least in this one study–it didn’t. Students who “elaboratively restudied” scored between the “pure restudy” group and the “retrieval practice” group–and their scores weren’t significantly different from either.
The Take-Aways…
I myself reach three conclusions based on this research:
A) Yup: retrieval practice still works, even with 4th graders, even with vocabulary learning, even in the Netherlands.
B) My instincts about elaborative restudy might be off. I should keep my eyes peeled for further research.
C) The muddle isn’t disheartening, it’s enjoyable. Jump in–the water’s warm!
Editor’s note: Dr. Agarwal will be speaking at next week’s Learning and the Brain conference. Here’s your chance to get to know her and her work better…
Andrew Watson:
I understand that you worked as a teacher before you started training as a scientist, so I’m curious about that experience. More specifically, how did your classroom experience shape your research interests?
Dr. Pooja K. Agarwal:
I completed Elementary Education Certification in undergrad, and I taught 4th and 5th grade. And throughout college I also developed curriculum and taught in summer camps: grades 3 through 12, mostly in science curriculum. So I was majoring in Elementary Education, and—of course—I was also taking Psychology courses.
It was fascinating to be taking my Education classes—where we’re learning pedagogy and teaching methods—simultaneously with taking Psychology classes with my mentor Roddy Roediger on the science of learning and cognitive psychology.
And the two fields just seemed completely disconnected. It was one of those “Aha!” moments for me where I didn’t understand why these two departments, and these two approaches, were so different—because they could really benefit from each other. And that’s what led to my combination of teaching—and my teaching approaches—and being a scientist.
Most of my research in the past 12 years has been in classrooms, as opposed to in laboratory settings. I really enjoy the messiness of doing scientific research in classrooms. The fire alarms, and school assemblies, and kids who are out sick, I really enjoy it because it pushes boundaries. We find all these great things about learning in the psychology laboratory with college students, but do they actually work in the real world?
Watson:
Your research focuses particularly on retrieval practice. Can you define that for us?
And, perhaps you could give an example of something that is retrieval practice, and something that isn’t: “If you do it this way, that’s retrieval practice. If you do it that way, it isn’t retrieval practice.”
Agarwal:
In a laboratory, in one of the comparisons we use most often, students read a passage: let’s say about sea otters.
We might have students read it once, and then have them write down everything they can remember about the passage three times. That would be retrieval practice. They had to bring the material about sea otters back to mind and retrieve what they learned.
On the other hand, after students read the passage, we might have them simply read it three more times. This rereading does not involve any retrieval.
Another example would be a student who re-reads their textbook; or who takes notes, and re-reads their notes; that’s not retrieval practice. As opposed to a student who uses flashcards: that’s retrieval practice.
What I like to say is: it’s the difference between trying to get information INTO your head by re-reading stuff over and over, versus trying to get information OUT of your head by using flashcards or writing down what you can remember.
Watson:
That’s a helpful and a clear way to explain the difference.
Agarwal:
Retrieval practice is actually more intuitive and used more often in K-12 than it is in college. In college, students are very used to sitting down, being talked at in a lecture, and then having to output things only on an exam. In K-12, I think we engage in retrieval much more often. There’s a lot more questioning, or cold calling, or discussion.
In K-12, we should be a little more mindful about, “If I’m going to review something as the teacher, can I change that to a retrieval activity?”
For instance, instead of saying, “Last week we learned about King Tut, and how old he was, and when he died, and what he did as Pharaoh,” a different angle to involve retrieval in a K-12 classroom would be: “All right, write down two things you remember from our discussion about King Tut.” And then have a brief discussion to remind everyone, “Oh, Andrew picked up on something that Pooja did not. And Emily remembered something that Thomas remembered as well. This is cool.”
That sort of retrieval activity could involve “think, pair, share,” where students are thinking, they’re writing down what they learned about King Tut, and then, of course, talking in pairs, and then sharing with the class.
It’s important to know that retrieval practice can take one minute or less, where students think about or write down what they remembered, and then you move on with your lesson – with or without discussion or grading. It’s also important to keep in mind that classroom retrieval activities should be low stakes. Make sure to emphasize retrieval as a learning strategy, not an assessment.
Watson:
You mentioned the difference between doing research in the lab—where most psychology research happens—and doing research in actual schools and classrooms. Can you talk more about that?
Agarwal:
A lot of research on retrieval practice has been in laboratory settings. Typically with lab research, all of the participants are college students. They come in, and we have them look at stuff on the computer: like foreign language word pairs, or brief reading passages: 500 words, maybe 1,000 words.
And then we distract them—we might have them play Tetris. And then, for instance, we ask students: “Okay, write down everything you remember from that passage about sea otters that you just read.”
And then we have students come back, let’s say a week later, and we ask them to do the same thing. “Hey, you read this passage about sea otters. It’s been a week. Your life went on. And now, write down everything you can still remember.”
And so we’ll ask: “How much do students remember a week later [after retrieval practice] compared to if they just re-read that passage over and over again without engaging in retrieval practice?”
And when it comes to classroom research, my colleagues and I have done similar research. We’ve played around with retrieval practice approaches and research questions like
Are multiple-choice quizzes versus short-answer quizzes more beneficial for learning?
Do online websites or retrieval programs like Kahoot! or Quia also boost student learning?
How many times do students have to retrieve in classroom settings to get the “biggest bang for your buck?”
Do pre-quizzes help, or is it better to quiz after a lesson?
Feedback: does it have to explain why students got something wrong, or simply indicate a correct or incorrect answer?
There are definitely advantages to extending lab research into classrooms. Not just to see if lab findings work in the real world, but there are some logistical things that are actually easier.
For instance, it’s hard to get college students to come back into a lab one week later. They forget. Even if we pay them and send email reminders, they don’t show up. They’re college students.
But in K-12, students go to school every day, and so we can look at rich learning in a real environment, and students come back a week later, or three months later, or seven months later. And the vast majority of students are still in school at the end of the school year. Laboratory research looking at learning and forgetting rarely goes beyond one week.
Watson:
Can you think of something in the world of retrieval practice that looked promising when it was researched in a lab, but when you tried it out in a classroom it just didn’t have the same effect there?
Agarwal:
Oh, that’s a good question. There are some lab findings that disappear in a classroom.
Watson:
I’m intrigued. “Disappear” sounds dramatic.
Agarwal:
Let’s look at multiple choice versus short answer retrieval in the form of low-stakes quizzes.
Based on 100 plus years of research, we’ve found that the more challenging the learning strategy, the more robust the learning over the long term. One researcher, Robert Bjork, coined the term “desirable difficulty.”
In line with that theory, from laboratory research it appeared that retrieval practice with short answer questions really boosted long term learning, compared to multiple-choice quizzes. Again, that seems pretty intuitive; there’s more of a desirable difficulty in writing a short answer as opposed to just being able to choose one of four multiple-choice options.
My colleagues (Kathleen McDermott, Roddy Roediger, and Mark McDaniel) and I did pretty much the exact same research in classrooms – 7th grade science and high school history. Students learned normal materials from their classroom teacher. Then we gave them either multiple-choice or short-answer quizzes. After a few days on a unit exam and even months later at the end of the semester, we saw a large benefit of retrieval practice, but the difference between short-answer quizzes and multiple-choice quizzesdisappeared.
It wasn’t as though short-answer quizzes helped students learn more than multiple-choice. That simple act of retrieval improved learning more than not engaging in retrieval at all.
For me that’s an example where findings from the lab don’t appear to apply in classroom settings.
And so I typically recommend to teachers, “You know what? Do what’s logistically easier.” Short-answer quizzes can take more time to grade. Of course, multiple-choice might take more time to develop, because you want to come up with really good alternatives. Either way, retrieval practice boosts long-term student learning, regardless of the low stakes quiz format.
Watson:
Is retrieval practice beneficial for all ages? Is it beneficial for all subjects that are taught?
Agarwal:
Patrice Bain has been a collaborator with me and my colleagues for more than 10 years. She and I like to say that retrieval practice works for all students, all subjects, all the time.
Our research in the Columbia School District outside of St. Louis in Illinois has included special ed. students, gifted students, students in pull-out tutoring programs. We still see the same benefits of retrieval practice.
In terms of all subjects: we’ve seen that retrieval benefits history, science, Spanish, vocabulary learning, a few others.
A subject area that we haven’t done too much research on is math, because—as you can imagine—in math they do retrieval all the time.
Watson:
Right, pretty much all math homework is retrieval practice. Does it work equally well for learning facts and learning skills?
Agarwal:
Good question. There’s more research on fact learning than skill learning, especially in laboratories.
In my dissertation, I focused on fact learning versus higher order learning. In one experiment, in the sixth-grade social studies classroom with my collaborator Patrice Bain, she was teaching chapters on World War II and the Russian Revolution.
I looked at how retrieval practice can improve fact learning—again, we already have lab and classroom research showing that it improves fact learning— sometimes even doubles learning compared to non-retrieval lessons.
But can we use higher order quizzes to improve higher order learning? For instance: questions that go beyond what is explicitly stated in the lesson.
I found that, excitingly, yes, especially with these complex materials, retrieval does improve higher order learning. (I based those materials on Bloom’s taxonomy.)
Watson:
Up to this point our conversation has focused primarily on psychology: which is to say, how minds work. I want to change gears and talk about neuroscience: which is to say how brains work. Do we, as researchers, have any understanding why retrieval practice helps new neural networks form to create long-term memories?
Agarwal:
A bit. That kind of work is recent, and so there’s still a lot more to be done.
The predominant research in neuroscience related to retrieval practice is about a process we call consolidation. The basic idea is that by retrieving, we’re recreating memories, and strengthening neural networks.
Exactly how that happens in terms of synapses or neurotransmitters is something we’re still trying to figure out. There’s not a whole lot known about it.
Watson:
That was my impression.
One of the points we emphasize on the blog is that, in the world of science, skepticism is key for what we do. Scientists know more today than they did yesterday because we’re a skeptical bunch.
I’m hoping that there is skepticism about retrieval practice, and I’m curious to know what you think is the most valid skepticism about it?
Agarwal:
Two things.
One of them is that retrieval practice is just a lot of tests: “Well, this just sounds like a lot of testing, and teaching to the test.”
Retrieval practice used to be referred to as “the testing effect.” Our field has moved away from that, especially because it’s not tests or quizzes that improve learning. It’s the actual process of retrieval. Retrieval is, in many ways, not even related to assessments.
The other main skeptical response is exactly what you asked about with fact learning versus higher-order learning, or more complex skills.
I agree that we should be a bit skeptical. There isn’t much research yet on extending retrieval practice to applied settings, let alone higher-order complex materials. Part of that reason, I think, is that in laboratory settings we like things really controlled so we can make sure A causes B. And when it comes to complex materials it’s very hard to do that.
For instance, it took me a year to develop the higher order complex materials for my dissertation.
I think that skepticism for higher order learning is warranted in that there isn’t yet much research yet. That being said, the research that is there—including my own research—shows that if we engage in complex skills during retrieval, then that will improve complex skills down the road.
Watson:
If I’m hearing you right, it sounds like the concept of retrieval practice itself is pretty well settled, and there aren’t people out there who say, “You did the math wrong. If you studied correctly, the effect goes away.”
Agarwal:
Correct. There is so much reliable research both for labs and in classrooms— especially from the past 15 years—showing that time and time again, huge effect sizes. I don’t think any skeptics still argue that we should be re-reading our textbooks instead of engaging in retrieval.
Watson:
Okay. Are there specific questions I should have asked that I haven’t yet asked you?
Agarwal:
Again, I’ll name two things.
One is: for more research, resources, and a down-loadable retrieval practice guide, go to retrievalpractice.org. We also send out weekly email updates with research summaries and evidence-based recommendations, too.
Second: I like to highlight what someone can do tomorrow. What can a teacher do tomorrow in their classroom to use this powerful strategy?
Depending on the subject area and the grade level, I like to highlight a technique called a “brain dump” which is simply asking students to write down everything they can remember about a class topic, lesson, unit, etc.
And that can take less than a minute. Also, teachers can include a “think, pair, share” after the brain dump or just move on.
So using retrieval practice doesn’t require redoing someone’s teaching approach, or curricula, or anything like that. It can just be as simple as, “Think about this question, or turn and talk to someone about what you learned.”
Watson:
Those are both very helpful.
One last question, which has nothing to do with science. A lot of people who will be attending the conference aren’t from Boston. Do you have any recommendations—restaurants, or museums, or pubs, or parks you think conference goers should see?
Agarwal:
I always recommend one thing that is never mentioned on tourist websites, called the Mapparium. I think it’s fantastic, and I’ve never seen something like it in any other city.
_____________________________
You can listen to an interview with Dr. Agarwal over at Cult of Pedagogy by clicking this link.
_____________________________
Biography:
Pooja K. Agarwal, Ph.D. is an expert in the field of cognitive science. She has conducted learning and memory research in a variety of classroom settings for more than 10 years. Passionate about evidence-based education, Pooja has extensive teaching experience in K-12 and higher education, as well as expertise in education policy at state and national levels. Currently, she is an Assistant Professor at the Berklee College of Music in Boston, teaching psychological science to exceptional undergraduate musicians.
To advance the use of scientifically-based learning strategies, Pooja founded RetrievalPractice.org, a hub of cognitive science research, resources, and tips for educators. Pooja’s research is supported by grants from the National Science Foundation and the U.S. Department of Education. In addition, her work has been featured in the New York Times, Education Week, and Scientific American, as well as academic journals, books, and podcasts. For more information, visit poojaagarwal.com and retrievalpractice.org.
