Tag: retrieval practice

Psychology can offer advice to teachers, but sometimes that advice is frustratingly vague.

We know, for example, that attention is important.

We know that it results from a combination of three neural processes: alertness, orienting, and executive attention.

But: what do teachers DO with that knowledge? How do we teach any differently?

Retrieval Practice Timing

For example, we’ve seen lots of research showing that retrieval practice helps students learn.

That is: rather than simply looking back over material, students should somehow quiz themselves on it.

They might use flash cards.

They might try a “brain dump”: just writing down everything they remember on a topic.

They can use quizlet to review key points.

Given that lots of student learning happens with textbooks, teachers have a very practical question: when should they do that retrieval?

Textbooks often include practice questions. Should students try to answer them as they go along? Or, should they wait until they have read the full chapter?

Happily, this question can be studied quite straightforwardly.

Uner & Roediger had three groups of students read a chapter from a science textbook.

One group did nothing to review.

A second group reread key passages from the book.

A third group answered the self-study questions in the textbook. Some of those questions appeared at the end of a section. Some appeared at the end of the chapter. And some appeared in both places.

Which group remembered this information better two days later?

Retrieval Practice Timing, and Beyond

Unsurprisingly, the students who reread the information remembered a bit more than those who did not. The rereaders averaged a 44.8 on the quiz, whereas the one-time readers averaged a 34.2.

The self-testers? They averaged a 61.5.

As we’ve seen before, self-testing is a HUGE help.

(By the way, you might think “61.5 is a terrible score. That’s almost failing!” However, this isn’t a class test; it’s a relative measurement. The point isn’t what the students scored, but how the groups scored compared to each other. The self-testers remembered much more.)

What about the timing? Is it better to answer questions at the end of the section, or the end of the chapter.

As it turned out, both times worked equally well. As long as students do the retrieval practice, it doesn’t particularly matter at what point in the chapter they do so.

Here’s the intriguing finding: questions answered twice — both at the end of the section and and the end of the chapter — led to even higher learning.

That might not sound surprising, but other researchers have found that one retrieval practice exercise produces as much benefit as two.

Keepin’ It Real

Psychology researchers could easily get focused on studies in the lab. They can control variables better; they’re faster to run.

I always feel especially happy to find researchers who keep their gaze on practical classroom applications.

In this case, we’ve learned: a) that retrieval practice helps students learn from textbooks, b) that students can answer relevant questions at any time and still get this benefit, and c) that two attempts to answer the question are (or, at least, might be) better than one.

Editor’s note: Dr. Agarwal will be speaking at next week’s Learning and the Brain conference. Here’s your chance to get to know her and her work better…

Andrew Watson:

I understand that you worked as a teacher before you started training as a scientist, so I’m curious about that experience. More specifically, how did your classroom experience shape your research interests?

Dr. Pooja K. Agarwal:

I completed Elementary Education Certification in undergrad, and I taught 4^th and 5^th grade. And throughout college I also developed curriculum and taught in summer camps: grades 3 through 12, mostly in science curriculum. So I was majoring in Elementary Education, and—of course—I was also taking Psychology courses.

It was fascinating to be taking my Education classes—where we’re learning pedagogy and teaching methods—simultaneously with taking Psychology classes with my mentor Roddy Roediger on the science of learning and cognitive psychology.

And the two fields just seemed completely disconnected. It was one of those “Aha!” moments for me where I didn’t understand why these two departments, and these two approaches, were so different—because they could really benefit from each other. And that’s what led to my combination of teaching—and my teaching approaches—and being a scientist.

Most of my research in the past 12 years has been in classrooms, as opposed to in laboratory settings. I really enjoy the messiness of doing scientific research in classrooms. The fire alarms, and school assemblies, and kids who are out sick, I really enjoy it because it pushes boundaries. We find all these great things about learning in the psychology laboratory with college students, but do they actually work in the real world?

Watson:

Your research focuses particularly on retrieval practice. Can you define that for us?

And, perhaps you could give an example of something that is retrieval practice, and something that isn’t: “If you do it this way, that’s retrieval practice. If you do it that way, it isn’t retrieval practice.”

Agarwal:

In a laboratory, in one of the comparisons we use most often, students read a passage: let’s say about sea otters.

We might have students read it once, and then have them write down everything they can remember about the passage three times. That would be retrieval practice. They had to bring the material about sea otters back to mind and retrieve what they learned.

On the other hand, after students read the passage, we might have them simply read it three more times. This rereading does not involve any retrieval.

Another example would be a student who re-reads their textbook; or who takes notes, and re-reads their notes; that’s not retrieval practice. As opposed to a student who uses flashcards: that’s retrieval practice.

What I like to say is: it’s the difference between trying to get information INTO your head by re-reading stuff over and over, versus trying to get information OUT of your head by using flashcards or writing down what you can remember.

Watson:

That’s a helpful and a clear way to explain the difference.

Agarwal:

Retrieval practice is actually more intuitive and used more often in K-12 than it is in college. In college, students are very used to sitting down, being talked at in a lecture, and then having to output things only on an exam. In K-12, I think we engage in retrieval much more often. There’s a lot more questioning, or cold calling, or discussion.

In K-12, we should be a little more mindful about, “If I’m going to review something as the teacher, can I change that to a retrieval activity?”

For instance, instead of saying, “Last week we learned about King Tut, and how old he was, and when he died, and what he did as Pharaoh,” a different angle to involve retrieval in a K-12 classroom would be: “All right, write down two things you remember from our discussion about King Tut.” And then have a brief discussion to remind everyone, “Oh, Andrew picked up on something that Pooja did not. And Emily remembered something that Thomas remembered as well. This is cool.”

That sort of retrieval activity could involve “think, pair, share,” where students are thinking, they’re writing down what they learned about King Tut, and then, of course, talking in pairs, and then sharing with the class.

It’s important to know that retrieval practice can take one minute or less, where students think about or write down what they remembered, and then you move on with your lesson – with or without discussion or grading. It’s also important to keep in mind that classroom retrieval activities should be low stakes. Make sure to emphasize retrieval as a learning strategy, not an assessment.

Watson:

You mentioned the difference between doing research in the lab—where most psychology research happens—and doing research in actual schools and classrooms. Can you talk more about that?

Agarwal:

A lot of research on retrieval practice has been in laboratory settings. Typically with lab research, all of the participants are college students. They come in, and we have them look at stuff on the computer: like foreign language word pairs, or brief reading passages: 500 words, maybe 1,000 words.

And then we distract them—we might have them play Tetris. And then, for instance, we ask students: “Okay, write down everything you remember from that passage about sea otters that you just read.”

And then we have students come back, let’s say a week later, and we ask them to do the same thing. “Hey, you read this passage about sea otters. It’s been a week. Your life went on. And now, write down everything you can still remember.”

And so we’ll ask: “How much do students remember a week later [after retrieval practice] compared to if they just re-read that passage over and over again without engaging in retrieval practice?”

And when it comes to classroom research, my colleagues and I have done similar research. We’ve played around with retrieval practice approaches and research questions like

• Are multiple-choice quizzes versus short-answer quizzes more beneficial for learning?
• Do online websites or retrieval programs like Kahoot! or Quia also boost student learning?
• How many times do students have to retrieve in classroom settings to get the “biggest bang for your buck?”
• Do pre-quizzes help, or is it better to quiz after a lesson?
• Feedback: does it have to explain why students got something wrong, or simply indicate a correct or incorrect answer?

There are definitely advantages to extending lab research into classrooms. Not just to see if lab findings work in the real world, but there are some logistical things that are actually easier.

For instance, it’s hard to get college students to come back into a lab one week later. They forget. Even if we pay them and send email reminders, they don’t show up. They’re college students.

But in K-12, students go to school every day, and so we can look at rich learning in a real environment, and students come back a week later, or three months later, or seven months later. And the vast majority of students are still in school at the end of the school year. Laboratory research looking at learning and forgetting rarely goes beyond one week.

Watson:

Can you think of something in the world of retrieval practice that looked promising when it was researched in a lab, but when you tried it out in a classroom it just didn’t have the same effect there?

Agarwal:

Oh, that’s a good question. There are some lab findings that disappear in a classroom.

Watson:

I’m intrigued. “Disappear” sounds dramatic.

Agarwal:

Let’s look at multiple choice versus short answer retrieval in the form of low-stakes quizzes.

Based on 100 plus years of research, we’ve found that the more challenging the learning strategy, the more robust the learning over the long term. One researcher, Robert Bjork, coined the term “desirable difficulty.”

In line with that theory, from laboratory research it appeared that retrieval practice with short answer questions really boosted long term learning, compared to multiple-choice quizzes. Again, that seems pretty intuitive; there’s more of a desirable difficulty in writing a short answer as opposed to just being able to choose one of four multiple-choice options.

My colleagues (Kathleen McDermott, Roddy Roediger, and Mark McDaniel) and I did pretty much the exact same research in classrooms – 7^th grade science and high school history. Students learned normal materials from their classroom teacher. Then we gave them either multiple-choice or short-answer quizzes. After a few days on a unit exam and even months later at the end of the semester, we saw a large benefit of retrieval practice, but the difference between short-answer quizzes and multiple-choice quizzes disappeared.

It wasn’t as though short-answer quizzes helped students learn more than multiple-choice. That simple act of retrieval improved learning more than not engaging in retrieval at all.

For me that’s an example where findings from the lab don’t appear to apply in classroom settings.

And so I typically recommend to teachers, “You know what? Do what’s logistically easier.” Short-answer quizzes can take more time to grade. Of course, multiple-choice might take more time to develop, because you want to come up with really good alternatives. Either way, retrieval practice boosts long-term student learning, regardless of the low stakes quiz format.

Watson:

Is retrieval practice beneficial for all ages? Is it beneficial for all subjects that are taught?

Agarwal:

Patrice Bain has been a collaborator with me and my colleagues for more than 10 years. She and I like to say that retrieval practice works for all students, all subjects, all the time.

Our research in the Columbia School District outside of St. Louis in Illinois has included special ed. students, gifted students, students in pull-out tutoring programs. We still see the same benefits of retrieval practice.

In terms of all subjects: we’ve seen that retrieval benefits history, science, Spanish, vocabulary learning, a few others.

A subject area that we haven’t done too much research on is math, because—as you can imagine—in math they do retrieval all the time.

Watson:

Right, pretty much all math homework is retrieval practice. Does it work equally well for learning facts and learning skills?

Agarwal:

Good question. There’s more research on fact learning than skill learning, especially in laboratories.

In my dissertation, I focused on fact learning versus higher order learning. In one experiment, in the sixth-grade social studies classroom with my collaborator Patrice Bain, she was teaching chapters on World War II and the Russian Revolution.

I looked at how retrieval practice can improve fact learning—again, we already have lab and classroom research showing that it improves fact learning— sometimes even doubles learning compared to non-retrieval lessons.

But can we use higher order quizzes to improve higher order learning? For instance: questions that go beyond what is explicitly stated in the lesson.

I found that, excitingly, yes, especially with these complex materials, retrieval does improve higher order learning. (I based those materials on Bloom’s taxonomy.)

Watson:

Up to this point our conversation has focused primarily on psychology: which is to say, how minds work. I want to change gears and talk about neuroscience: which is to say how brains work. Do we, as researchers, have any understanding why retrieval practice helps new neural networks form to create long-term memories?

Agarwal:

A bit. That kind of work is recent, and so there’s still a lot more to be done.

The predominant research in neuroscience related to retrieval practice is about a process we call consolidation. The basic idea is that by retrieving, we’re recreating memories, and strengthening neural networks.

Exactly how that happens in terms of synapses or neurotransmitters is something we’re still trying to figure out. There’s not a whole lot known about it.

Watson:

That was my impression.

One of the points we emphasize on the blog is that, in the world of science, skepticism is key for what we do. Scientists know more today than they did yesterday because we’re a skeptical bunch.

I’m hoping that there is skepticism about retrieval practice, and I’m curious to know what you think is the most valid skepticism about it?

Agarwal:

Two things.

One of them is that retrieval practice is just a lot of tests: “Well, this just sounds like a lot of testing, and teaching to the test.”

Retrieval practice used to be referred to as “the testing effect.” Our field has moved away from that, especially because it’s not tests or quizzes that improve learning. It’s the actual process of retrieval. Retrieval is, in many ways, not even related to assessments.

The other main skeptical response is exactly what you asked about with fact learning versus higher-order learning, or more complex skills.

I agree that we should be a bit skeptical. There isn’t much research yet on extending retrieval practice to applied settings, let alone higher-order complex materials. Part of that reason, I think, is that in laboratory settings we like things really controlled so we can make sure A causes B. And when it comes to complex materials it’s very hard to do that.

For instance, it took me a year to develop the higher order complex materials for my dissertation.

I think that skepticism for higher order learning is warranted in that there isn’t yet much research yet. That being said, the research that is there—including my own research—shows that if we engage in complex skills during retrieval, then that will improve complex skills down the road.

Watson:

If I’m hearing you right, it sounds like the concept of retrieval practice itself is pretty well settled, and there aren’t people out there who say, “You did the math wrong. If you studied correctly, the effect goes away.”

Agarwal:

Correct. There is so much reliable research both for labs and in classrooms— especially from the past 15 years—showing that time and time again, huge effect sizes. I don’t think any skeptics still argue that we should be re-reading our textbooks instead of engaging in retrieval.

Watson:

Okay. Are there specific questions I should have asked that I haven’t yet asked you?

Agarwal:

Again, I’ll name two things.

One is: for more research, resources, and a down-loadable retrieval practice guide, go to retrievalpractice.org. We also send out weekly email updates with research summaries and evidence-based recommendations, too.

Second: I like to highlight what someone can do tomorrow. What can a teacher do tomorrow in their classroom to use this powerful strategy?

Depending on the subject area and the grade level, I like to highlight a technique called a “brain dump” which is simply asking students to write down everything they can remember about a class topic, lesson, unit, etc.

And that can take less than a minute. Also, teachers can include a “think, pair, share” after the brain dump or just move on.

So using retrieval practice doesn’t require redoing someone’s teaching approach, or curricula, or anything like that. It can just be as simple as, “Think about this question, or turn and talk to someone about what you learned.”

Watson:

Those are both very helpful.

One last question, which has nothing to do with science. A lot of people who will be attending the conference aren’t from Boston. Do you have any recommendations—restaurants, or museums, or pubs, or parks you think conference goers should see?

Agarwal:

I always recommend one thing that is never mentioned on tourist websites, called the Mapparium. I think it’s fantastic, and I’ve never seen something like it in any other city.

_____________________________

You can listen to an interview with Dr. Agarwal over at Cult of Pedagogy by clicking this link.

_____________________________

Biography:

Pooja K. Agarwal, Ph.D. is an expert in the field of cognitive science. She has conducted learning and memory research in a variety of classroom settings for more than 10 years. Passionate about evidence-based education, Pooja has extensive teaching experience in K-12 and higher education, as well as expertise in education policy at state and national levels. Currently, she is an Assistant Professor at the Berklee College of Music in Boston, teaching psychological science to exceptional undergraduate musicians.

To advance the use of scientifically-based learning strategies, Pooja founded RetrievalPractice.org, a hub of cognitive science research, resources, and tips for educators. Pooja’s research is supported by grants from the National Science Foundation and the U.S. Department of Education. In addition, her work has been featured in the New York Times, Education Week, and Scientific American, as well as academic journals, books, and podcasts. For more information, visit poojaagarwal.com and retrievalpractice.org.

Contact Information

Pooja K. Agarwal, Ph.D.

Assistant Professor

Berklee College of Music

[email protected]

@PoojaAgarwal

poojaagarwal.com

@RetrieveLearn

retrievalpractice.org