Skip to main content
To Grade or Not to Grade: Should Retrieval Practice Quizzes Be Scored?
Andrew Watson
Andrew Watson

We’ve seen enough research on retrieval practice to know: it rocks.

When students simply review material (review their notes; reread the chapter), that mental work doesn’t help them learn.

However, when they try to remember (quiz themselves, use flashcards), this kind of mental work does result in greater learning.

In Agarwal and Bain’s elegant phrasing: don’t ask students to put information back into their brains. Instead, ask them to pull information out of their brains.

Like all teaching guidance, however, the suggestion “use retrieval practice!” requires nuanced exploration.

What are the best methods for doing so?

Are some retrieval practice strategies more effective?

Are some frankly harmful?

Any on-point research would be welcomed.

On-Point Research

Here’s a simple and practical question. If we use pop quizzes as a form of retrieval practice, should we grade them?

In other words: do graded pop quizzes result in more or less learning, compared to their ungraded cousins?

This study, it turns out, can be run fairly easily.

Dr. Maya Khanna taught three sections of an Intro to Psychology course. The first section had no pop quizzes. In the second section, Khanna gave six graded pop quizzes. In the third, six ungraded pop quizzes.

Students also filled out a questionnaire about their experience taking those quizzes.

What did Khanna learn? Did the quizzes help? Did grading them matter?

The Envelope Please

The big headline: the ungraded quizzes helped students on the final exam.

Roughly: students who took the ungraded pop quizzes averaged a B- on the final exam.

Students in the other two groups averaged in the mid-to-high C range. (The precise comparisons require lots of stats speak.)

An important note: students in the “ungraded” group scored higher even though the final exam did not repeat the questions from those pop quizzes. (The same material was covered on the exam, but the questions themselves were different.)

Of course, we also wonder about our students’ stress. Did these quizzes raise anxiety levels?

According to the questionnaires, nope.

Khanna’s students responded to this statement: “The inclusion of quizzes in this course made me feel anxious.”

A 1 meant “strongly disagree.”

A 9 meant “strongly agree.”

In other words, a LOWER rating suggests that the quizzes didn’t increase stress.

Students who took the graded quizzes averaged an answer of 4.20.

Students who took the ungraded quizzes averaged an answer of 2.96.

So, neither group felt much stress as a result of the quizzes. And, the students in the ungraded group felt even less.

In the Classroom

I myself use this technique as one of a great many retrieval practice strategies.

My students’ homework sometimes includes retrieval practice exercises.

I often begin class with some lively cold-calling to promote retrieval practice.

Occasionally — last Thursday, in fact — I begin class by saying: “Take out a blank piece of paper. This is NOT a quiz. It will NOT be graded. We’re using a different kind of retrieval practice to start us off today.”

As is always true, I’m combining this research with my own experience and classroom circumstances.

Khanna gave her quizzes at the end of class; I do mine at the beginning.

Because I’ve taught high school for centuries, I’m confident my students feel comfortable doing this kind of written work. If you teach younger grades, or in a different school context, your own experience might suggest a different approach.

To promote interleaving, I include questions from many topics (Define “bildungsroman.” Write a sentence with a participle. Give an example of Janie exercising agency in last night’s reading.) You might focus on one topic to build your students’ confidence.

Whichever approach you take, Khanna’s research suggests that retrieval practice quizzes don’t increase stress and don’t require grades.

As I said: retrieval practice rocks!

What (and Why) Should Students Memorize? Confidence and Fluency for the Win
Andrew Watson
Andrew Watson

In our profession, memorization has gotten a bad name.

The word conjures up alarming images: Dickensian brutes wielding rulers, insisting on “facts, facts, facts!”

In a world when students “can look up anything on the interwebs,” why do we ask students to memorize at all?

One answer from cognitive science: we think better with information we know well.

Even when we can find information on the internet, we don’t use that information very effectively. (Why? Working memory limitations muddle our processing of external information.)

A second answer appears in intriguing recent research.

Reasonable Plans, Unstated Assumptions

As a classroom teacher, I might operate on this reasonable plan:

Step one: we discuss ideas and information in class.

Step two: students write down the important parts.

And, step three: when students need that information later, they look at their notes.

This plan — the core of most high school classes I know — relies on unstated assumptions:

Assumption one: students’ notes are largely correct.

Assumption two: if students write down information INcorrectly, they’ll recognize that mistake. After all, we discussed the correct information in class.

But what if that second assumption isn’t true?

What if students trust external information (their notes) more than internal information (their memories)?

Assumptions Thwarted

In 2019, Risko, Kelly, & Gaspar studied one version of this question.

They had students listen to word lists, and type them into a storable file. After distraction, students got to review their lists. They then were tested on those words.

On the final list, however, these scholars did a sneaky thing: they added a word to the stored list. Sure enough, 100% of their students wrote down the additional word, even though it hadn’t in fact been on the initial word list.

Students trusted their written document (external “memory”) more than their own actual memory. When tested even later, students still included the additional word, even though it wasn’t a part of their initial learning.

In other words: the “reasonable plan” that teachers often rely on includes an assumption that — at least in this research — isn’t true.

Ugh.

Classroom Implications

This research, I think, reminds us that the right kind of memorization has great value for students.

We want students to know certain bedrock facts and processes with absolute certainty. We want them, for instance, to define key terms and ideas fluently. Crucially, we want them to reject — with confidence borne of certain knowledge — inaccurate claims.

For instance:

I just completed a unit on tragedy. My sophomores read August Wilson’s Fences and Shakespeare’s Macbeth.

 

On the very first day of the term, I went over a four-part definition of “tragedy.” (It’s a quirky definition, I admit, but it works really well.)

 

We reviewed that definition almost daily, increasingly relying on retrieval practice. For instance, I might give them this handout to fill in. Or we might do that work together on the board.

Over time, I started including inaccurate prompts in my questions: “So, tragedy ends in death or marriage, right?”

By this point, my students knew the definition so well that they confidently rejected my falsehoods: “No, you’re trying to trick us! Tragedy ends in death or banishment!”

For an even trickier approach, I encouraged students to correct one another’s (non-existent) mistakes:

Me: “T: what does comedy represent, and why?”

T: “The marriage (and implied birth) at the end of a comedy implies the continuity of society, and in that way contrasts tragedy’s death and banishment, which represent the end of society.”

Me: “M: what did T get wrong.”

M [confidently]: “Nothing. That was exactly right.”

Me [faking exasperation]: “S, help me out here. What did T and M miss?”

S [not fooled]: “Nothing. I agree with them both.”

Me: “Congrats to T for getting the answer just right. And congrats to M and S for not letting me fool you. It’s GREAT that you’re all so confident about this complex idea.”

Because these students knew this complex definition cold — because they had memorized it — they could stand firm when questioned skeptically. As a result, they did a great job when asked to apply that definition at the end of the term:

“How does Wilson’s Fences fit the definition of tragedy AND of comedy?”

To Sum Up

Despite all the bad press, the right kind of memorization can enhance learning.

When students know foundational information and processes by heart, they

BOTH process questions more fluently

AND resist misleading information from “external memory” sources.

Greater cognitive fluency + greater confidence in their knowledge = enduring learning.

Prior Knowledge: Building the Right Floor [Updated]
Andrew Watson
Andrew Watson

When I first published this post back in 2019, the underlying research wasn’t publicly available; now it is!


Take a gander at this passage from Michael Jones’s recent biography of The Black Prince:

“In the fourteenth century England used a silver standard of currency. The unit of account was the pound sterling (£) which was equal to one and a half marks of silver. The pound was divided into twenty shillings (s), each of twelve pence (d). There was also, from 1344, a gold coinage based on the noble, which was conventionally worth 6s 8d, but was rarely used. It would, however, be significant in the calculation of the ransom of King John II and also in the introduction of gold coinage into Gascony and then the principality of Aquitaine by the Black Prince.”

Many readers, I suspect, felt tempted to give up relatively quickly. (Don’t blame yourself if you did.) Unless you’re really up to speed on 14th century English currency–both silver and gold!–the paragraph quickly becomes overwhelming.

The vocabulary in this passage probably doesn’t strain our cognition. Except for the phrase “marks of silver,” I know what all those words mean. (And, I can guess from context that a “mark” is some unit of measurement.)

However, the passage does place several mental demands on the reader.

First, it invites you to undertake several rapid mathematical calculations. (Quick: how many shillings in a mark?)

Second, it requires you to learn abbreviations as you go. To understand the fourth sentence, you need to remember the (wildly counter-intuitive) abbreviation of “pence” as “d” from the third sentence.

Third, it assumes you recall several events and places unfamiliar–I suspect–to most Americans. Who was King John II? Why was he ransomed…was he kidnapped? Where are Gascony and Aquitaine? They don’t sound very English — why did an English prince introduce coinage to them? Actually: why is a prince empowered to introduce new currency?

Essential Background Knowledge

I thought of this paragraph when I read a recent article by Robert Pondiscio. In it, Pondiscio summarizes a study trying to determine how much background knowledge is necessary for comprehension.

In this study, students who scored higher than a 59% on a background knowledge test understood a reading passage substantially better than those who scored below 59%.

As summarized by Pondiscio, the study’s authors see some clear teaching implications here.

First, we can meaningfully measure our students’ relevant background knowledge.

Second, students who fall short on that measure will benefit A LOT if we provide them with the essentials.

For instance, students who understood that “habitat,” “species,” and “ecosystems” were relevant vocabulary for the study of ecology understood the reading passage more deeply. (The study included 3500 students, so I believe they controlled for various confounds. I haven’t read the study itself–it’s behind a paywall.)

I think those conclusions point to another:

Third: models of teaching that focus on “pure discovery” will create substantial challenges for students who lack background knowledge. Students who don’t know the basics of a topic simply can’t understand the field of inquiry within which they’re meant to discover.

And, they won’t feel motivated by curiosity to find out. They’ll feel discouraged by their confusion. (Few readers, I suspect, were motivated by the paragraph above to learn more about medieval English currency.)

A Final Thought

This study finds that 59% was the essential tipping point. Students who scored lower than 59% on the prior knowledge test found themselves in a different cognitive category than those who scored above.

Howeverthat percentage does not necessarily apply to all circumstances.

In other words: we shouldn’t give our students prior-knowledge tests, and focus only on those who score 58% and below.

Instead, we should plan our lessons and units knowing that some floor-level of knowledge will be crucial for learning most things.

In every case–as you hear me say so often–we’ll have to rely on the teacher’s judgment to discover that level.

Researchers can remind us that the floor exists. But they can’t identify it for every teacher in every classroom. Ultimately, with that research guidance in mind, we’ll find the right place for the floor. And, we’ll build it.

Game on? Brain On!: The Surprising Relationship between Play and Gray (Matter) by Lindsay Portnoy
Rebecca Gotlieb
Rebecca Gotlieb

Game on? Brain On!: The Surprising Relationship between Play and Gray (Matter)  is an affectionate, evidence-based, tribute to the importance of play for learning and preparing young people for their future. Author Lindsay Portnoy, who currently serves as an Associate Teaching Professor at Northeastern University, argues that we are born to play and that games can be an ideal space to develop skill in solving problems, thinking critically and creatively, persevering, collaborating, communicating, and empathizing. Portnoy argues for the value of play for supporting each individual student’s learning and for addressing equity issues in education. Throughout the book and on her website she offers tools to support educators in transitioning to more play-based learning exercises for students and urges educators to push back against rigid, putative learning cultures and standardized assessments. Especially amidst the COVID-19 pandemic when students miss their friends and may find online learning to be exhausting and when teachers are seeing anew how challenging standardized curricula can be, we could better serve our students by absorbing Portnoy’s lessons about how to harness the power of play. As such, Game On? Brain On!  will be of interest to an array of K-16 educators and education reformers.

Drawing on neurodevelopmental research Portnoy first argues that people come into the world ready to learn. Play is a natural way of facilitating learning by capitalizing on our intrinsic interests in a low-stakes, imaginative context. Portnoy offers numerous examples of games that students enjoy playing and can be educational—from commercially available board games (e.g., Monopoly), to video games (e.g. Fortnite), to physical games at recess (e.g., wall ball), to student or teacher invented games. By observing and probing students about their play while letting the students take the lead, educators can learn a lot about those students’ skills and interests. Simultaneously, they can support the students in developing critical skills such as executive functioning, emotional regulation, convergent and divergent thinking, and metacognition. Among the benefits of play are that it does not ask of students the same high degree of conformity and compliance that classrooms typically do and it draws on students’ strengths rather than admonishing them for their weaknesses.

Classic psychological motivation theories support Portnoy’s call for more game play. While playing games students can experience a sense of competence, autonomy, and connectedness with peers. Games allow students and their playmates to try on different roles creating a safe way to express different parts of themselves, opportunities to draw on peers’ expertise as sources of support and guidance, and a scaffolded context for building empathy. Cognitive psychologists and learning scientists know well that distributed, interleaved practice of skills supports long-term retention. Games encourage recursive practice, and they frame failure as an opportunity for feedback and growth, which supports learning.

Although educators do not often use games as a tool for assessment, games can be effectively harnessed as a way for students to demonstrate what they know and how they can continue to grow. Portnoy urges educators to consider employing reflection about experiences with games or other active learning exercises as a mode of assessment.

Citing examples of several games that are addressing pressing issues, Portnoy notes that through games students may be able to make authentic contributions to real problems in their community or in our society. Further, in school we often divide students by age or ability, but games are conducive to having players of all different ages, interests, and abilities work together, which can offer important social learning opportunities for students.

Children will always play.  If we paid less attention to what students play and learn and more attention to how they play and learn, we might be able to bring out the best in our learners by capitalizing on their passions to inspire their continued growth.

Portnoy, L. (2020). Game on? Brain On!: The surprising relationship between play and gray (matter). Dave Burgess Consulting, Inc.

Assembling the Big Classroom Picture
Andrew Watson
Andrew Watson

The last 20 years have brought about powerful new ways to think about teaching and learning.

When teachers combine our experience, professional traditions, and instincts with the scientific insights of psychology and neuroscience research, we find new ways to understand our classroom work.

As I’ve written before, we currently suffer from an embarrassment of riches.

With SO MANY ideas that have strong and consistent research backing, how can we understand and prioritize all the suggestions coming our way? (One recent answer: Chew and Cerbin’s approach, focusing on the cognitive challenges students face.)

The Big 6

You’d like another way to think about all this research?

Over at The Learning Scientists, several wise scholars have been championing a blend of six research-based approaches to teaching: spacing, interleaving, retrieval practice, elaboration, concrete examples, and dual coding.

Frequent readers will recognize almost every strategy on that list. I write about at least one of those each month. (If you want more information on “retrieval practice,” for instance, click on the tag in the right column.)

Two of the Learning Scientists, Dr. Yana Weinstein and Dr. Megan Sumeracki, have written a great book: Understanding How We Learn, a Visual Guide. If you want to explore those big 6 in depth, you should put it on your “must read” pile right away.

If, however, you’d like a brisker introduction to these ideas, I’ve recently found a review article by Weinstein and Sumeracki (and Christopher Madan) that summarizes and illustrates all six.

Newcomers to this field you can read its fifteen pages in an hour.

Veterans can instead scan it for a refresher, and for new insights and combinations.

For Example

Although I’ve written frequently about most of the Big Six, I don’t often discuss elaboration. More than the other concepts, it resists easy definition.

Luckily — more than most psychology terms (I’m looking at you, “theory of mind”) — the term means what it says. When we ask students to elaborate on new information or new skills, this deeper processing helps consolidate long-term memories.

For instance, in preparation for teaching Their Eyes Were Watching God, I took some time in Thursday’s class to define the terms “voice” and “agency” as English scholars use them. These concepts will be essential to understand Janie’s development over the course of the novel.

We then elaborated on those definitions by discussing two plays we had just finished reading: Fences and Macbeth.

When does Lady Macbeth have “voice”?

How does Troy claim “agency”?

What strategies does Roes use to claim voice and agency in her life?

How does the set of Fences represent limitations on agency?

And so forth.

By connecting these two new words to events they already knew and understood, students elaborated on their definitions. (And — as a bonus — got some retrieval practice about Fences and Macbeth.)

By the way, the Enser’s handy book Generative Learning in Action offers LOTS of well-researched elaboration strategies.

In Sum

At Learning and the Brain, we bring teachers and researchers together for productive conversation.

Sometimes, we should focus on research details to ensure they’re persuasive enough for us to reshape our teaching practices.

At other times, we should refocus on the big picture — to understand how each of the techniques adds up to a coherent teaching practice.

Weinstein, Madan, and Sumeracki help us do just that.

Enjoy!

Retrieval Practice and Metacognition: What and How Do Students Think about This Powerful Learning Strategy?
Andrew Watson
Andrew Watson

Ask almost anyone in Learning and the Brain world, they’ll tell you: retrieval practice benefits students.

More than most any other technique we have, this one both has lots of research support and can easily be integrated into our classrooms. (For a handy review of its benefits, check out Agarwal and Bain’s great bookPowerful Teaching.)

Of course, because retrieval practice offers so many benefits, we want our students to use it.

Do they?

Metacognition and Retrieval Practice

The short answers are: not enough, and not very well.

Michelle Rivers wanted to know why, and so explored research into students’ metacognitive beliefs about retrieval practice. That is:

What do they believe about retrieval practice?

How do they use it to monitor their learning?

How do they use its insights to control their learning activities?

The more we understands students’ metacognitive perspective on their learning, the more wisely we can guide them.

What did she find?

Beliefs about Retrieval Practice

Sadly, most students don’t understand RP’s benefits.

In 10 studies that asked “why do you quiz yourself,” only 26% of students say they do so in order to learn more.

Instead, most students (52%) do so “to figure out how well I’ve learned the information I’m studying.”

In other words: even the students who use RP most often do so for the wrong reasons.

Of course: they’re not harming themselves by using retrieval practice this way. But — and this is a big but — they’re not getting the benefits that RP can offer.

In fact, Rivers’s survey suggests one reasons students might not use retrieval practice to help themselves learn. Studies suggest that when students try both methods, they don’t predict that they’ll remember more after retrieval practice. (Check out this study by Kornell and Son.)

I find this research pool baffling, even disheartening. Even when students experience greater success with RP than with simple rereading, they don’t internalize the broader lesson that active retrieval helps them learn.

Little wonder, then, that most students review material (43%) or copy their notes (11%) as a go-to strategy, rather than self-testing (8%).

Uses of Retrieval Practice

Given these flawed beliefs, how do students use RP?

Among Rivers’s findings: students try retrieval practice …

… when the questions are easy

right before a test

relatively late in the learning process.

… relatively few times for any given pool of information.

Of course, retrieval practice benefits students when they do so…

… with questions that are challenging

well before a test (in fact, RP immediately before a test might be counterproductive)

throughout the learning process

several times for any given pool of information.

Simply put: even when students use this excellent study strategy, they do so in less-than-optimal ways.

Next Steps: Learning How to Learn

So far, this is quite the glum post. A potentially powerful learning strategy is largely going to waste.

What can we teachers do?

I’ve got two suggestions.

First, this recent post summarizes a promising approach from Mark McDaniel and Gilles Einstein. Their multi-step process not only works to persuade students of RP’s benefits; it encourages them to make specific retrieval practice plans and to follow through on them.

In other words: we shouldn’t just tell our students about its benefits. We shouldn’t just tell them to do it. We should go the next steps to create plans and structures.

Second, I’ve seen LOTS of online programs to help teachers and students with their retrieval practice.

For instance, Adam Boxer has created a program called Carousel. This program allows teachers to create retrieval questions, and to jumble them together in useful ways. It allows students to self-score their work (and teachers to correct any errors). It keeps track of right and wrong answers, so we can see how well our students are learning specific topics and questions.

I have used Carousel enough to find it intriguing; I haven’t used it enough to make strong claims about it. (Given responses to it on Twitter, however, it seems that teachers kind of love it.)

Whichever program you choose, I think students will learn how to learn more effectively if we build tools like these into our teaching practice.

In Sum

A: We know that retrieval practice can help students learn, but only if they use it correctly.

B: We know that, for the most part, they don’t.

A + B = We should focus more on helping students use this strategy wisely. And, we’ve got the tools to do so!

 

“Kids These Days!”: A (Partial) Defense of Ignorance and Distractibility
Andrew Watson
Andrew Watson

You’ve seen the videos.

An earnest reporter wielding a microphone accosts a college student and asks extremely basic questions:

“What are the three branches of government?”

“What is the capital of France?”

“Who wrote the Declaration of Independence?”

When students can’t answer these questions, the reporter eyes the camera wryly, as if to say, “What’s wrong with Kids These Days?”

One such video made the rounds recently. Middle schoolers (I think) didn’t know what D-Day was: they hypothesized he might be a rapper.

So, really: what is wrong with these kids? How can they POSSIBLY not know about D-Day?

Beyond History

Our lament frequently goes beyond students’ lack of historical knowledge.

We worry about “kids and their devices.”

They’re always looking at screens! (I’m here to tell you: back in the ’80s, we never looked at screens.)

They’re always texting! (We never texted.)

They’re so distractible! (Nope. Not us.)

If students know so little and concentrate so badly, we have to wonder what’s up with them.

Distateful Mirrors

I understand the frustration. (I’ve taught a class of well-educated students who didn’t know the story of Noah’s Ark. That was a shocker.)

At the same time, I think that question distracts us from the essential underlying point.

If an entire room of students didn’t know what D-Day was, it’s clear that the school system didn’t teach them about D-Day; or — at least — that didn’t teach them well enough for them to consolidate that knowledge.

If we can easily find college students who don’t know from geography and history, we can blame the college students. But I think we should first pause to consider the education system in which it’s possible to complete high school without enduring knowledge of such things.

It is, in my view, simply not fair or helpful to blame students for being in the system that adults created.

Those videos shouldn’t make us condemn the students; they should instead make us look in the mirror.

We might not like what we see. But: their shortcomings tell us more about our education system than about our students.

The Ingenious Tech Switcheroo

The same argument, in my view, applies to laments about technology. Notice the impressive blame-shifting here:

Step 1: technology companies invent a must-have gadget. They market it to — say — 10 year olds.

Step 2: 10-year-olds want the gadget, and pester their parents to buy it. (The tech company doesn’t need to get money from adults; they persuade children to get money from their parents and give it to the company. BRILLIANT!)

Step 3: the tech industry then highlights the narrative that the 10-year-olds are to blame for being distractible. The problem is not in the adults’ behavior; it’s in the children! “Why oh why can’t these kids focus!”

Here again, the students’ behavior gives us essential feedback about adults.

If we want today’s students to concentrate better, maybe we should create — and provide them with — fewer distractions. Perhaps we should model the behavior we want to see. (Quick: how many browser tabs do you have open?)

Caveats, Always Caveats

One: not everyone worries when students don’t know stuff.  (I do, but some don’t share that concern.) For adults who don’t emphasize factual knowledge, those videos seem trivial, not alarming.

Two: not all the data suggest that students are “more distractible.” Perhaps they simply have more distractions. (How many times did you check your cell phone in the last hour?)

Three: Of course, students bear responsibility for working effectively within the systems adults create. If twenty-four of my students learn something and one doesn’t, we can reasonably wonder what’s going on with that one. But, if none of my students know the importance of the Treaty of Versailles, I should think about the adult-driven systems, not “what’s up with kids these days.”

Four (this is a biggie): As I strive refocus popular outrage away from the students and toward the system in which they learn, I might seem to be blaming teachers and school leaders. I very much don’t mean to do that.

In my experience, the great majority of both groups work extremely hard, and do so with the best intentions. (Few people say: “I went into teaching to become rich.”)

At the same time: our well-intentioned efforts simply aren’t producing the results we want. That feedback — evident in those videos — should prompt honest and searching self-reflection.

In Sum

I promised a (partial) defense of ignorance and distractibility. Here goes:

Of course we want our children to know important information and skills, and to be able to concentrate on them.

If most students don’t and can’t, the fault probably lies with the education system, not children who learn within it.

Children who don’t know what D-Day is don’t deserve to be ridiculed on Twitter. They do deserve a curriculum that fosters knowledge, skill, and concentration. They deserve pedagogy that helps them master all three.

At Learning and the Brain, we connect education with psychology and neuroscience in order to start conversations. Conversations that include those three perspectives can help create such a curriculum; can help foster such pedagogy.

We hope you’ll join us!