Skip to main content

Andrew Watson About Andrew Watson

Andrew began his classroom life as a high-school English teacher in 1988, and has been working in or near schools ever since. In 2008, Andrew began exploring the practical application of psychology and neuroscience in his classroom. In 2011, he earned his M. Ed. from the “Mind, Brain, Education” program at Harvard University. As President of “Translate the Brain,” Andrew now works with teachers, students, administrators, and parents to make learning easier and teaching more effective. He has presented at schools and workshops across the country; he also serves as an adviser to several organizations, including “The People’s Science.” Andrew is the author of "Learning Begins: The Science of Working Memory and Attention for the Classroom Teacher."

Early Thoughts on A.I. Research in Schools
Andrew Watson
Andrew Watson

I hope that one of my strengths as a blogger is: I know what I don’t know — and I don’t write about those topics.

While I DO know a lot about cognitive science — working memory, self-determination theory, retrieval practice — I DON’T know a lot about technology. And: I’m only a few miles into my own A.I. journey; no doubt there will be thousands of miles to go. (My first foray along the ChatGPT path, back in February of this year, did not go well…)

A young child types on a laptop; a small robot points out answers on a see-through screen that hovers between them

Recently I came across research that looks at A.I.’s potential benefits for studying. Because I know studying research quite well, I feel confident enough to describe this particular experiment and consider its implications for our work.

But before I describe that study…

Guiding Principles

Although I’m not a student of A.I., I AM a student of thinking. Few cognitive principles have proven more enduring than Dan Willingham’s immortal sentence: “memory is the residue of thought.”

In other words, if teachers want students to remember something, we must ensure that they think about it.

More specifically:

  • they should think about it successfully (so we don’t want to overload working memory)
  • they should think about it many times (so spacing and interleaving will be important cognitive principles
  • they should think hard about it (so desirable difficulty is a thing)

And so forth.

This core principle — “memory is the residue of thought” — prompts an obvious concern about A.I. in education.

In theory, A.I. simplifies complex tasks. In other words, it reduces the amount of time I think about that complexity.

If artificial intelligence reduces the amount of time I that I’m required to think about doing the thing, it necessarily reduces the amount of learning I’ll do about the thing.

If “memory is the residue of thought,” then less thinking means less memory, and less learning…

Who Did What?

Although discussions of generative A.I. often sound impenetrable to me, this study followed a clear and sensible design.

Researchers from the University of Pennslyvania worked with almost 1000 students at a high school in Turkey. (In this kind of research, 1000 is an unusually high number.)

These students spent time REVIEWING math concepts they had already learned. This review happened in three phases:

Phase 1: the teacher re-explained math concepts.

Phase 2: the students practiced independently.

Phase 3: the students took a test on those math concepts. (No book; no notes; nada.)

For all students, phases 1 and 3 were identical. Phase 2, however, gave researchers a chance to explore their question.

Some students (let’s call them Group A) practiced in the usual way: the textbook, their notes, paper and pencil.

Group B, on the other hand, practiced with ChatGPT at hand. They could ask it questions to assist with their review.

Group C practiced with a specially designed ChatGPT tutor. This tutor was programed not to give answers to students’ questions, but to provide hints. (There were other differences between the ChatGPT and the ChatGPT tutor, but this difference strikes me as most pertinent.)

So: did ChatGPT help?

Did the students in Groups B and C have greater success on the practice problems, compared to Group A?

Did they do better on the test?

Intriguing Results

The students who used A.I. did better on the practice problems.

Those who used ChatGPT scored 48% higher than their peers in Group A.

Those who used the ChatGPT tutor scored (are you sitting down?) 127% higher than their peers in Group A.

Numbers like these really get our attention!

And yet…we’re more interested in knowing how they did on the test; that is, how well did they do when they couldn’t look at their books, or ask Chatty questions.

In brief: had they LEARNED the math concepts?

The students who used regular ChatGPT scored 17% lower than their notes-n-textbook peers.

Those who used the ChatGPT tutor scored the same as those peers.

In brief:

A.I. helped students succeed during practice.

But, because it reduced the amount of time they had to THINK about the problems, it didn’t help them learn.

Case closed.

Case Closed?

In education, we all too easily rush to extremes. In this case, we might easily summarize this study in two sentences:

“A.I. certainly didn’t help students learn; in some cases it harmed their learning. Banish A.I.!”

While I understand that summary, I don’t think it captures the full message that this study gives us.

Yes: if we let students ask ChatGPT questions, they think less and therefore learn less. (Why do they think less? Probably they simply ask for the answer to the question.)

But: if we design a tutor that offers hints not answers, we reduce that problem … and eliminate the difference in learning. (Yes: the reseachers have data showing that the students spent more time asking the tutor questions; presumably they had to think harder while doing so.)

As a non-expert in this field, I suspect that — sooner or later — wise people somewhere will be able to design A.I. tutors that are better at asking thought-provoking hints. That is: perhaps an A.I. tutor might cause students to think even MORE than other students praticing the old-fashioned way.

That two sentence summary above might hold true today. But we’ve learned this year that A.I. evolves VERY rapidly. Who knows what next month will bring.

TL;DR

Although THIS study suggests that A.I. doesn’t help (and might harm) learning, it also suggests that more beneficial A.I. tutors might exist in the future.

If — and this is the ESSENTIAL “if” — if A.I. can prompt students to THINK MORE than they currently do while practicing, then well-established cog-sci principles suggest that our students will learn more.


* A note about the publication status of this study. It has not yet been peer reviewed and published, although it is “under review” at a well-known journal. So, it’s technically a “working paper.” If you want to get your research geek on, you can check out the link above.


Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, O., & Mariman, R. (2024). Generative ai can harm learning. Available at SSRN4895486.

Teachers’ Professionalism: Are We Pilots or Architects?
Andrew Watson
Andrew Watson

I recently attended a (non-Learning-and-the-Brain) conference, and saw a thoughtful presentation that included a discussion of teachers’ professional standing.

In this blog post, I want to …

  1. summarize this speaker’s thoughtful argument,
  2. explain my own reasons for doubting it, and
  3. consider some weaknesses in my own argument.

Teachers as Pilots

The speaker started by summarizing a common claim about teachers’ professional independence:

“Because teachers are highly-trained professionals, we should have the same freedom for independent action and creativity that other professionals enjoy. Rather than scripting and constraining teachers, schools should allow them the leeway to think, act, and teach with meaningful independence.”

I should be clear, by the way, that this speaker’s summary is NOT a straw man. I know that people make (roughly) this argument because a) I’ve heard other people make it, and b) I’m going to make a similar argument in just a few paragraphs.

To interrogate this pro-independence argument, the speaker asks us to think about other highly esteemed professionals: doctors, airline pilots, and engineers.

In every case, these professionals work in highly constrained conditions. In fact, we would be rightly shocked were they to want to escape those constraints. Imagine:

  • If a pilot were to say: “today I think it would be fun to go through this pre-flight check list in reverse order. And, heck, I think I’ll skip step #27; I’ve never understood what it was for!”
  • If an ER doctor were to say: “I understand that you’re experiencing chest pains, and the protocols suggest several tests. But honestly: you look healthy to me. And, I’m feeling lucky today. So let’s assume it’s a touch of nerves and get you some nice chamomile tea.”

We would be horrified!

Pilots and doctors work within well-established constraints, and have a strict ethical obligation to follow them.

For this reason, society rightly condemns instances where these professionals go outside those constraints.

A woman seated in a small airplane cockpit with her hands on the yoke

Another example: when engineers get sloppy and bridges fall down mid-construction, we feel both horror and surprise to learn that they didn’t follow professional codes that govern their work.

These examples — the speaker said — show that teachers’ demands for professional freedom are misplaced.

Tight constraints do not violate our professional standing; they embody our professional standing.

Pushing Back: Reason #1

Although I understand these arguments as far as they go, I disagree with the speaker’s conclusion. Let me see if I can persuade you.

I think that doctors, pilots, and engineers are not good analogues for teachers, because the systems on which doctors, pilots, and engineers operate are unified, coherent, and designed to function as a whole.

Here’s what I mean:

  • An airplane’s ailerons and flaps have been scrupulously designed to act upon the wings in a specific way. So too the engines and the rudders. And, frankly, almost everything else about the airplane.

Because airplane parts have been structured to function together, we can have specific, precise, and absolute rules the operation of planes. When the flaps do this, the airflow over the wing changes in entirely predictable ways. The plane will, in these circumstances, always turn, or ascend, or land, or whatever.

Yes, special circumstances exist: turbulence, wind shear, or thunderstorms. But even these special circumstances call for predictable and consistent responses: responses that can be trained, and should be executed precisely.

  • A bridge has been designed to balance the forces that act upon it and the materials used to build it. Steel does not wake up one day and decide to have the strength of aluminum. Gravity does not vary unpredictably from day to day or mood to mood.

Because bridges have been structured to function in a particular way, engineers can have specific, precise, and absolute rules about their construction. Engineers don’t insist on moment-by-moment freedom because the bridges they build have entirely predictable constraints.

If, however, you have worked in a classroom, you know that such absolute predictability – based on the unity of the system being operated – has almost nothing to do with a teacher’s daily function.

  • Gravity doesn’t work differently before and after lunch, but students do.
  • An airplane’s rudder doesn’t have a different response to the pilot’s input, but this student might have a very different response than that student to a teacher’s input.
  • An EKG (I assume) shows a particular kind of result for a healthy heart and a distinct one for an unhealthy heart. A student’s test results might mean all sorts of things depending on all sorts of variables.
  • By the way: all of these examples so far focus on one student at a time. They don’t begin to explore the infinite, often-unpredictable interactions among students…
  • …or the differences among the topics that students learn…
  • …or the cultures within which the students learn.

We shouldn’t treat a classroom system as a straightforwardly stimulus-response system (like an airplane, like a bridge) because classrooms include an unpredictable vortex of possibilities between stimulus and response.

The best way – in many cases, the ONLY way – to manage that vortex: give teachers professional leeway to act, decide, change, and improvise.

The Continuum of Professionalism

Let’s pause for a moment to consider other kinds of professionls — say, ARCHITECTS, or THERAPISTS.

We allow — in fact we expect — subsantial freedom and creativity and variety as they do their work.

Of course, these professionals work within some constraints, and follow a well-defined code of ethics.

But those modest contraints allow for great freedom because…

… this client wants her house to look like a Tudor castle, while that client wants his to look like Falling Water, or

… this client struggles with PTSD and doesn’t want to take meds, while that client is managing bipolar disorder.

In other words: some professions do permit — in fact, require — strict limitations. But not all professions do. And, in my view, ours doesn’t. We’re more like architects than engineers.

Pushing Back: Reason #2

I promised two reasons that I resist the call for doctor-like-narrow-constraints. Here’s the second.

The analogies provided in this case all focus on people dying. The plane crashed. The heart-attack patient perished in agony. The bridge crushed workers and passers by.

In these professions, constraints literally save lives.

Now, I (like all teachers) think that education is important, and can transform lives and societies for the better. Bad educational practices do have damaging results for individuals and communities.

But: no one ever died from a bad English class. (I know; I’ve taught bad English classes. My students didn’t learn much, but they survived.)

If, in fact, teachers should work within tight constraints — checklists, scripts, step-by-step codes — the argument in favor of that position should be persuasive without the threat of death to energize it.

I’m Right, but I Might be Wrong

I promised up top that I’d include the weaknesses in my argument. I see at least four.

One:

Obviously, novice teachers require lots of support, and should work within tighter constraints than experienced teachers.

And, some teachers aren’t very good at their jobs. We might reasonably decline to trust their professional judgments.

Also: some people LIKE scripted curricula. I don’t think we should take them away from people who want them.

Two:

Teachers shouldn’t be scripted or managed detail by detail, but we should operate within well-established cognitive science principles. For instance:

  • Retrieval practice is, in almost all cases, better than simple review
  • Working memory overload is, in ALL cases, a detriment to learning
  • Myths like “learning styles” or “right/left brain learning” should not be a guide for instruction.

In other words: my call for independence isn’t absolute. We should know our subject and our craft — and then work flexibly and appropriately with those knowledge bases.

Three:

I suspect that a few specific disciplines might allow for precise scripting.

The teaching of reading or early math, for instance, might really benefit from doing EXACTLY step A and then EXACTLY step B, and so forth.

However:

Even in this case, I suspect we would need LOTS of different scripts:

… “typical” readers,

… students with dyslexia, or diagnosably low working memory,

… students with unusually low background knowledge,

… students whose home language isn’t the instructional language,

and so forth.

In other words: even a scriptable subject matter requires teacherly expertise and inventiveness in moving from script to script.

Four:

My own biases no doubt shape my argument.

I myself am a VERY independent person, and I have a penchant for holding teachers in great esteem.

For these reasons, I probably react more strongly than others do to the suggestion that teachers should be tightly constrained to meet our professional obligations.

In other words: I do have a logical argument in support of my position.

a) Flying an airplane is an inappropriate analogue for teaching a class;

b) Tight constraints are almost certainly impossible in the work we do.

But: that logical argument almost certainly gets much of its passion from a realm beyond logic.

In Sum

Ideas about pedagogy often rest on assumptions about teacher independence. For that reason, I’m glad that the speaker raised this point specifically, and made a strong argument to support one point of view.

I hope I’ve persuaded you that teachers — like architects — need informed independence (within some constraints) to do our work.

Even if not, I’m hopeful that thinking through these questions has proven helpful to you.


 

In this tweet threat, the invaluable Peps Mccrea gives an excellent example of a situation where teachers’ communal adherence to the same norms benefits students and schools.

The Benefits (and Perils) of Thinking Hard
Andrew Watson
Andrew Watson

Back in 2010, Professor Dan Willingham launched a movement with his now-classic book Why Don’t Students Like School?

In that book — one of the first to make cognitive science clear and practical for classroom teachers — Willingham wrote this immortal sentence: “Memory is the residue of thought.”

A triangular yellow road sign reading "Hard Work Ahead." the sign has a large hand print on it, and smudges of dirt or grime.

In other words: if we want students to learn something — and we do! — we need to make them think hard about it.

When I first read that sentence, it sounded almost too obvious to write down … until I recalled all the things that we (and I) do that DON’T require students to think about the topic we want them to learn.

Once we take this insight onboard, all sorts of other ideas start making more sense.

For instance: why is retrieval practice better at creating long-term memories than simple review?

Well, because students have to think harder when they retrieve than when they review. And because they think harder, they remember more. (“Memory is the residue of thought.”)

Simply put: we teachers strive to foster as much focused, hard thought as possible.

Voila — teaching made easy!

Not So Fast…

This core advice offers such practical insights: what could possibly be the problem?

Well, imagine this hypothetical:

If our students believe that fruits and vegetables are bad for their health, they face two compelling reasons not to eat ’em:

  • vegetables taste bad (I’m sorry, I don’t get kale), and
  • vegetable are bad for health.

That second belief might be wrong. But if the students believe it, it will influence their thinking and behavior.

Let’s translate this hypothetical to classrooms.

If our students believe that hard thinking is proof that they didn’t learn, they face two compelling reasons not to think hard:

  • thinking hard is unpleasant, and
  • thinking hard is proof that students don’t get the concept.

Now, you and I know that this second belief is false. Hard thinking is NOT proof that students don’t get the concept; it is — quite the contrary — an essential step on the way to getting the concept.

Hard thinking is — as the technology people say — not a bug; it’s a feature.

For these reasons, we really want to know: do our students hold this false belief? Do they think that hard thinking is both unpleasant and a sign of cognitive ineffectiveness?

In a word: yup.

Let’s Get Technical

A recent meta-analysis looks at precisely this question.

In the highly technical language of research, they conclude:

“The amount of mental effort experienced during a learning task is usually negatively correlated with learners’ perception of learning.”

In other words: just as we feared, students (on average) perceive mental effort to be unpleasant AND a sign that they’re not learning.

In other words:

“If a learner judges their perceived  mental effort to be high, they will judge their perceived learning…as low.”

In 1984, this situation would be described as “double-plus-un-good.”

As is usually the case, the methodology for this meta-analysis involves technical yada-yada that’s more complicated than is worth exploring. But those headlines should be clear enough to get our teacherly attention.

Translating Research for the Classroom

What should school people do with this information?

I’ve got three suggestions:

First:

“Just tell them.” *

That is:

  • Specifically say to students that, although it seems PLAUSIBLE that hard thought is a sign of academic failure,
  • Hard mental work actually helps them learn.

This truth feels obvious to adults, but is counter-intuitive to younger learners.

Of course, we will need to make this point over and over. I regularly say to my students: “That feeling in your head right now…that feeling ‘this is really complicated!’ … that feeling is you learning the concept.

Second:

Explain to students — in an age appropriate way — that research supports this claim.

For instance:

You could ask your students: “which is more difficult: a) trying to remember information, or b) rereading it?”

You can then show them LOTS O’ RESEARCH showing that trying to remember — a.k.a. “retrieval practice” — is MUCH more beneficial for learning.

Third:

Depending on the student, an analogy might be helpful: specifically a sports or arts analogy.

In my experience, most students accept without question that sports and arts require repetitious hard work.

  • If you want to get better at your dance steps, repeat your dance steps
  • If you’re bad at hitting 3-point shots, practice 3 point shots
  • If you hope to feel more natural with the blocking at the top of act II, you know what to do.

In every case, the need for all this hard work isn’t a sign that you can’t dance/score/act; instead, it’s a perfectly normal part of getting better at dancing, scoring, and acting.

By the way, I should probably include a fourth strategy:

“You will know what works best for you and your students.”

As is so often the case when translating research for the classroom, the teacher’s perspective often provides the best ideas.

Once you and your classroom colleagues know that students mis-interpret hard thought to be a bad thing, you will start coming up with the solutions likeliest to update this misconception.

Research can help us spot problems; we teachers are often even better at developing effective solutions.


 

* Yes, I’m slyly quoting the title of my friend Zach Groshell’s book.


David, L., Biwer, F., Baars, M., Wijnia, L., Paas, F., & De Bruin, A. (2024). The relation between perceived mental effort, monitoring judgments, and learning outcomes: A meta-analysis. Educational Psychology Review36(3), 66.

 

Learning Goals Reconsidered (No, Not THOSE Learning Goals)
Andrew Watson
Andrew Watson

I’ve been discussing a topic with colleagues in recent months, and want to share my thinking with you.

The outline for this blog post is:

An observation/question that has been nagging at me, then

A theory about the answer to the question, then

Possible implications of that theory.

Here goes:

A Puzzling Problem

In recent years, as I listen to discussions on BIG EDUCATIONAL TOPICS, I frequently find myself surprised by this truth:

Education is full of folks who are

  1. Obviously smart,
  2. Obviously well informed,
  3. Obviously concerned about their students, our profession, and our society, and
  4. FEROCIOUSLY, ANGRILY at odds with one another.

Seriously, if folks could punch one another online, Twitter would have LOTS of broken noses.

This observation leads to a straightforward question: why? Why do these ferocious disagreements persist?

A man and woman sit across a small table talking with each other. He shrugs his shoulders in puzzlement, she points in irritation.

If many of us think ably and know a lot and care deeply, it’s surprising that we disagree…and keep disagreeing. The heated arguments just don’t change much.

In many scientific fields, heated arguments result – over time, at least – in the ultimate persuasiveness of one case or another.

Plate tectonics used to be a controversial topic, with lots of heated rhetoric. Today, those debates among smart, knowledgeable, and caring people have resulted in something like consensus: the theory is almost certainly correct.

Surgeons now wash their hands before surgery. That practice was initially scorned, but now the anti-handwashing argument seems impossible to understand. OF COURSE one side prevailed in the debate.

Why haven’t educational debates followed this pattern? Why haven’t our disagreements led ultimately to agreements?

One Possibility

I’ve been discussing a possible answer with several friends: here goes.

I wonder if our debates remain so heated because we don’t agree on the GOALS of education. In fact, we aren’t even in the habit of discussing those goals – or the habit of connecting them to teaching practices.

Off the top of my head, I can imagine all sorts of big-picture goals for the millions of dollars and millions of hours our society devotes to creating and maintaining its educational system.

If I were to ask 100 people this question, I can imagine a wide range of answers: “the goal of our educational system is to…

  1. Create a workforce for the future,” or
  2. Help students understand and enact the word of God,” or
  3. Know the best that has been thought or said,” or
  4. Ensure that individual students develop to their fullest potential,” or
  5. Promote justice, equity, and peace,” or
  6. Preserve our culture and way of life,” or
  7. Raise scores on key benchmark assessments,” or
  8. Prepare children and society for an unpredictable future,” or
  9. Develop students who see themselves as readers, historians, scientists, etc.,” or
  10. Ensure that students know the curriculum,” or
  11. Create an informed and civic-minded electorate,” or
  12. Foster a love of learning so that students become life-long learners,” or …

…and so forth.

Perhaps the reason our debates about teaching strategies go nowhere is that we’re in fact trying to go different places.

That is:

I might read about a teaching strategy and think “Pish posh. That’s obviously bonkers. It simply won’t accomplish the core aims of education.”

And yet, the person proposing the strategy might well have entirely different aims. And – sure enough – the teaching strategy being proposed might achieve their core aims, if not mine.

If, for example, I practice the Awesome Watson Learning Method, I might to do because it fosters a love of learning (goal 12) and ensures that students see themselves as writers and programmers and doctors (goal 9).

A critic might respond: “that pedagogy won’t accomplish our goal!” And that criticism might be sincere, because the pedagogy doesn’t (let’s say) help students “learn the greatest that has been thought or said” (goal 3). Yet because I’m not striving for goal 3, I’m genuinely vexed and puzzled by my critic’s (obviously incorrect) critique.

Humbling Implications

If I’m right that our “debates” simply talk past one another because we don’t share — or discuss — educational goals, that realization suggests several next steps.

Step A:

The next time I hear someone espouse a teaching method that strikes me as foolish, I should switch from contempt to curiosity.

Rather than “pish posh,” I should say: “That’s intriguing — tell me more!”

If I ask the right questions in an open-minded, non-snarky way, I might discover an entirely unexpected goal at the end of the process. I might not agree about the importance of that goal, but I might …

…understand why the other person champions it, and

…recognize that the teaching strategy I once thought foolish might in fact accomplish it.

Sadly, this “switch from contempt to curiosity” is really difficult. I will face the constant temptation to ask leading questions and trap my interlocutors into admitting that my goal surpasses theirs in wisdom and beauty.

(The best book I’ve read that discusses this problem is David McRaney’s How Minds Change. It has really shaped my thinking on this challege.)

Step B:

Since 2008, I’ve been thinking about using scientific research — especially in psychology and neuroscience — to improve my teaching.

Obviously, this approach focuses on numerical measurements: calculations, graphs, statistics.

In other words: I believe that my teaching strategies accomplish my goals because I’ve got numbers that say so.

However, several of the big-picture goals listed above simply can’t be measured.

How would I know if the Awesome Watson Teaching Method…

… helps students become life-long learners?

… ultimately fosters civic engagement?

… encourages students to live and act according to God’s word?

The end point for these goals (and others) lies decades away — and will be influenced by THOUSANDS of other forces.

This fact, however, does not necessarily invalidate the potential importance of those goals.

Teachers might not be able to show a study — with p-values in the appropriate range, and a Cohen’s d above 0.2 — concluding that their teaching method promotes justice and peace. But that impossibility does not mean that their goal has no merit.

In other words: I’m attracted to a science-y approach to thinking about teaching practice, and I like being able to cite all those numbers. (92% of in-classroom studies show that  retrieval practice promotes long-term memory better than control conditions!)

But science-y approaches can’t routinely dictate answers to moral or ethical questions.

Another Possibility?

Of course, I have a MUCH simpler explanation for the fact that many people disagree with me — often angrily:

Those other people could be daft, ignorant, and/or immoral.

That explanation has several benefits.

  • It’s easy to summarize.
  • It converts me from a person into a hero/protagonist.
  • It frees me from the need to listen to their foolish, ill-informed, morally-tainted ideas.

At the same time, I find this simpler explanation unsatisfying — because I disagree with many people who don’t strike me as daft or wicked.

Perhaps there’s a third explanation?

TL;DR

I’m trying to focus less on why others are wrong. I’m trying to focus more on their implied goals for education — goals that have led them to teaching advice that puzzles or alarms me.

When I understand their goals, I might better understand — and learn from — their teaching suggestions.

Perhaps you’ll join me in this effort — and let me know what you learn.


 

In case the title of this post doesn’t make sense: researchers in the world of mindset encourage a less focus on performance goals (test scores, etc.) and more focus on learning goals (“look! I made progress!”).

This blog post isn’t about mindset-y learning goals, but about society’s broader goals for education.

Incremental Steps with Growth Mindset
Andrew Watson
Andrew Watson

The field of education often races to extremes, and the field of Growth Mindset has been an especially good example of this trend.

Back in the 2006 when Carol Dweck published her book, schools rushed to be as Growth Mindset-y as possible. Posters adorned walls; Janelle Monaie made a Sesame Street video reminding children about “the Power of ‘Yet’.”

Little Asian boy with stadiometer near green wall

All those enthusiasts felt quite a shock in 2018, when two mega-meta-analysis crunched all the numbers and found that

a) mindset doesn’t really have much of an effect on academic performance, and

b) all those mindset interventions don’t really do anything anyway.

In some academic quarters, loud praise gave way to ridicule and — in some cases — snark. (The loaded claim that “all the research showing a positive effect was done by Dweck herself” simply isn’t true.)

Since then, competing camps wave conflicting studies to support their pro/anti Growth Mindset position.

I’d like to advocate for an alternative approach. I believe Dr. Dan Willingham said something like this:

“Some studies suggest that a growth mindset helps some students; other studies suggest that creating the conditions to develop and enact that mindset is REALLY TRICKY.

We shouldn’t simply abandon this approach. We should focus our efforts on finding when it does and doesn’t help which students, and how to foster those conditions for them.”

In other words: a growth mindset won’t promptly and easily cure all motivation problems. But let’s try to find where and how and whom it benefits.

With that goal in mind, I want to explore a recent study. My goal is NOT to say “this team is right; that one is wrong.”

Instead, I want to show how this study gives us reasons to be hopeful and curious — but should still not return us to the days of A Poster in Every Classroom.

Best Case Scenario

In this study from 2022, roughly 80 children aged 7-10 had 2 fMRI scans separated by four weeks.

They also filled out a mindset survey, agreeing or disagreeing with statements like “I can get better at math if I work hard to solve problems,” or “people are born smart or not smart, and there’s not much they can do to change that.”

For half of those children – the control group – that was that.

The other children — during the intervening four weeks — went through a specially designed math tutoring program.

This tutoring program emphasized progress and understanding, not simply scores or comparisons to others. If you know from growth mindset, you know the terminology here: the program emphasized “mastery/learning goals,” not “performance goals.”

So, what did the research team find after four weeks?

Several important findings jump out from all the data and charts:

First:

BOTH GROUPS saw an increase in their growth mindset “score.” For instance, they were likelier to disagree with the statement that “people can’t do much to change their math ability.”

However – and this is an important however – the GROUP IN THE TUTORING PROGRAM saw a bigger change. If you think having a growth mindset is good, you’d interpret these data to say that the tutoring group “made more progress.”

In other words: contrary to that big-news meta-analysis from 2018, this study found that — under these conditions — growth mindset training did help students develop more of a growth mindset.

Second:

We care about mindset because it should motivate students to learn more. To say the same thing in different words: if students who do have a growth mindset learn as much as those who don’t, why would we focus so much energy on developing that mindset?

The research team wanted to know if students who had a more of a growth mindset before the tutoring program learning more math during the tutoring program.

The technical answer to this question is: “yup.”

Third:

When the research team compared changes in fMRI scans after four weeks, they found that the changes in growth mindset correlated with specific changes in neural regions and networks.

If you want to get your neuro-geek on: in the scans of children with higher mindset scores, they found

  • greater activation at the top of the front of the cingulate cortex (“dorsal ACC”)
  • greater activation in the top of the right striatum
  • greater activation in the right hippocampus

They also found that changes in the circuitry connecting these regions “emerged as the strongest predictor of growth mindset gains.”

Recapping the Best Case

Yes: we have had reasonable doubts about the importance of mindset. (Reasonable doubts = that 2018 meta-analysis, among other studies.)

But, this study arrives at three striking conclusions:

a) a well-designed math tutoring program can foster a growth mindset,

b) growth mindset before the tutoring program results in greater math learning during that program, and

c) we see brain regions both activating and connecting differently in conjunction with growth mindset self-report.

Even for skeptics, that’s an impressive combination!

Pushing Back

I can see at least two reasons this study isn’t a slam dunk for Team Mindset.

Reason A:

When researchers compare two groups — as they did in this case — we want those groups to be as much alike as possible.

While these groups did resemble each other in lots of important ways (average age, average mindset score, average IQ, etc.), they differed in a crucial one: one group got something; the other group got nothing.

That is: one group received four weeks of tutoring. The control group simply went about “business as usual.”

They did not — for instance — get a different kind of tutoring that did NOT focus on a growth mindset.

We can therefore wonder if these students developed a growth mindset not because they got a special kind of tutoring, but because they got any tutoring. Maybe the tutoring, not the mindset ingredients, made the difference.

Reason B:

If you read this blog often, you know that I’m very wary of people who make strong claims based on neuroscience research.

Lots of people make claims like these: “when teachers do X, students get a hit of oxytocin. So everyone has to X!”

Here’s my argument: until we test X with actual students doing something in actual classrooms, we don’t know whether or not extra oxytocin does anything beneficial in these circumstances. (Yes, someone is ACHING to call oxytocin the “love hormone.” No, it’s really not.)

So, can we think of other reasons these students’ brain structures and networks might have changed?

Here’s a possibility: perhaps their brains responded to extra math tutoring.

Because they had different experiences for four weeks, it’s not wholly shocking that their brains developed differently from those in the control group.

In other words: just because this study includes complicated brain terminology does NOT mean that we must be persuaded by its conclusions. LOTS of people make strong claims about neuroscience; not all of them hold up well.

(To be fair: an earlier study with college students found the dorsal ACC to be an important part of the growth mindset network. This reduplication clearly makes the current neuro-claim more plausible.)

A Final Verdict

Now that I’ve made arguments both championing and questioning this study, you might reasonably want a clear answer.

Rather than provide false certainty, I’ll go a different direction.

As Dan Willingham (I think) said: we’re trying to figure out where, when, and with whom mindset interventions work.

Based on this study, we can say: “it seems likelier that youngsters given a particular kind of tutoring develop more of a growth mindset; it also seems likely that this mindset helps them learn math.”

That’s not a universal claim; it’s quite a narrow one.

To develop a more complete and persuasive understanding, we will need all sorts of incremental steps:

One research group will work with 5th graders in science classes.

Another will focus on the neuroscience of mindset in high-stress situations.

A third will turn its attention to adults who return to school to pursue a second career.

And so on.

Piece by piece, study by study, we will gradually accumulate a clearer mental model. In a few decades, we will probably be saying: “we used to talk about mindset in this crude, outdated, and puzzling way. But now that we understand this mental phenomenon so much better, we know that…”

And the advice that follows will be granular, targeted, and perhaps surprising to us who got our start making mindset posters.


Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L., & Macnamara, B. N. (2018). To what extent and under which circumstances are growth mind-sets important to academic achievement? Two meta-analyses. Psychological science29(4), 549-571.

Chen, L., Chang, H., Rudoler, J., Arnardottir, E., Zhang, Y., de Los Angeles, C., & Menon, V. (2022). Cognitive training enhances growth mindset in children through plasticity of cortico-striatal circuits. npj Science of Learning7(1), 30.

Mangels, J. A., Butterfield, B., Lamb, J., Good, C., & Dweck, C. S. (2006). Why do beliefs about intelligence influence learning success? A social cognitive neuroscience model. Social cognitive and affective neuroscience1(2), 75-86.

Even More Questions (3rd of a Series)
Andrew Watson
Andrew Watson

This blog post continues a series about research into questions.

I started with questions that teachers should ask BEFORE students’ learning begins: “pre-questions,” measuring prior knowledge.

I then turned to questions that we ask DURING early learing: retrieval practice, checking for understanding.

Now — can you guess? — I’ll focus on questions that we ask LATER in learning, or “AFTER” learning.

To structure these posts, I’ve been focusing on three organizing questions:

When to ask this kind of question? (Before, during/early, during/later)

Who benefits most immediately from doing so?

What do we do with the answers?

Let’s dive in…

A Controversy Resolved?

At some point, almost all teaching units come to an end. When that happens, teachers want to know: “how much did my students learn?”

To find out, we typically ask students questions. We might call these questions “quizzes” or “tests” or “assessements” or “projects.”

A young girl reads and draws in a garden

Whatever we call such questions, students answer by writing or saying or doing something.

Who benefits from all these activities? Well, here we arrive at a controversy, because reasonable people disagree on this point.

OVER HERE, some folks argue that assessments basically benefits school systems — and harm others. After assessments, school systems can…

  • sort students into groups by grade, or
  • boast about their rising standardized test scores, or
  • evaluate teachers based on such numbers.

I don’t doubt that, in some cases, assessments serve these purposes and no others.

OVER THERE, more optimistically, others argue that assessments can benefit both teacher and student.

Students benefit because

  • They learn how much they did or didn’t learn: an essential step for metacognition; and
  • The act of answering these questions in fact helps students solidify their learning (that’s “retrieval practice,” or “the testing effect”).

Teachers benefit because

  • We learn how much our teaching strategies have helped students learn, and
  • In cumulative classes, we know what kinds of foundational knowledge our students have for the next unit. (If my students do well on the “comedy/tragedy” project, I can plan a more ambitious “bildungsroman” unit for their upcoming work.)

In other words: final assessments and grades certainly be critiqued. At the same time, as long as they’re required, we should be aware of and focus on their potential benefits.

Digging Deeper

While I do think we have to understand the role of tests/exams/capstone projects at the “end” of learning, I do want to back up a step to think about an intermediate step.

To do so, I want to focus on generative questions — especially as described by Zoe and Mark Enser’s excellent book on the topic.*

As the Ensers describe, generative questions require students to select, organize, and integrate information — much of which is already stored in long-term memory.

So:

Retrieval practice: define “bildungsroman.”

Generative learning: can a tragedy be a bildungsroman?

The first question asks a student to retrieve info from long-term memory. The second requires students to recall information — and to do mental work with it: they organize and integrate the parts of those definitions.

For that reason, I think of retrieval practice as an early-in-the-learning-process question. Generative learning comes later in the process — that is, after students have relevant ideas in long-term memory to select, organize, and integrate.

The Ensers’ book explores research into, and practical uses of, several generative learning strategies: drawing, mind-mapping, summarizing, teaching, and so forth.

In my thinking, those distinct sub-categories are less important that the overall concept. If students select, organize, and integrate, they are by definition answering generative learning questions.

For instance: the question “can a tragedy be a bildungsroman” doesn’t obviously fit any of the generative learning categories. But because it DOES require students to select, organize, and integrate, I think it fits the definition.

(I should fess up: technically, retrieval practice is considered a generative learning strategy. For the reasons described above, I think it’s helpful to use RP early in learning, and generative learning later in learning. My heresy could be misguided.)

“Generative learning” is a BIG category; teachers can prompt students to think generatively in all sorts of ways. A recent review by Garvin Brod suggests that some strategies work better than others for different age groups: you can check out those guidelines here.

TL;DR

In most school systems, teachers must ask some kind of summary questions (tests, projects) at the end of a unit. Such questions — if well designed — can benefit both teachers and students.

After students have a bedrock of useful knowledge and before we get to those final test/project questions, teachers should invite students to engage in generative learning. By selecting, organizing, and reintegrating their well-established knowledge, students solidify that learning, and make it more flexible and useful.


Brod, G. (2021). Generative learning: Which strategies for what age?. Educational Psychology Review33(4), 1295-1318.


* Grammar nerds: if you’re wondering why I wrote “Zoe and Mark Enser’s book” instead of “Zoe and Mark Ensers’ book” — well — I found that apostrophe question a stumper. I consulted twitter and got emphatic and contradictory answers. I decided to go with the apostrophe form that makes each Enser and invidivual — because each one is. But, I could be technically wrong about that form.

The Best Way to Teach: When Clarity Leads to Muddle
Andrew Watson
Andrew Watson

Most teachers want to be better teachers. You’re probably reading this blog for research-based guidance on doing so.

A young student wearing plastic goggles carefully pours something into a beaker slightly filled with green liquid

I recently read a study that offers emphatic — and paradoxical — guidance. Exploring this research — as well as its paradoxes — might be helpful as we think about being better teachers.

Here’s the story.

A research team, led by Louis Deslauriers, worked with students in an introductory physics class at Harvard. This class was taught by an experienced professor who mostly lectured; he also supplemented the class with “demonstrations, … occasional interactive quizzes or conceptual questions.”

Let’s call this approach “interactive lecture.”

In Deslauriers’s study, students also attended two additional classes. One was taught with Method A and the other with Method B.

In Method A, an experienced professor:

  • presented slides
  • gave explanations
  • solved sample problems
  • strove for fluency of presentation

What abotu Method B? Another experienced teacher:

  • used principles of deliberate practice
  • instructed students to solve sample problems together in small groups
  • circulated through the room to answer questions
  • ultimately provided a full and correct answer to the problems

The researchers strove, as much as possible, to make the class content identical; only the pedagogy differed.

What did the researchers learn about the relative benefits of Methods A and B?

Paradox #1: Effective and Unloved

First off, the students learned more from Method B.

That is: when they solved problems in small groups, wrestled with the content, and ultimately heard the right answer, students scored relatively higher on an end-of-class multiple choice test. When they experienced Method A (the prof explained all the info and solved all the problems), they scored relatively lower.

But — paradoxically — the students preferred Method A, and believed that they learned more from it. They even suggested that all their classes be taught according to Method A — the method that resulted in less learning.

The researchers offer several explanations for this paradox. The headlines sound like this:

  • When students hear straightforward explanations and see clear/succesful demonstrations of solutions strategies (Method A), the content seems easy and clear. Students think they understand.
  • But, when they have to do the cognitive heavy lifting (Method B), class feels more difficult. Result: students worry they didn’t understand.
  • Because the students are — relatively speaking — novices, they don’t know enough to know when they understand.

Team Deslauriers, sensibly enough, suggests that we can help students appreciate and accept the more challenging methods — like Method B — if we explain the reseasoning behind them.

I, by the way, take this suggestion myself. For instance: I explain the benefits of retrieval practice to my students. They don’t always love RP exercises, because retrieval practice feels harder than simple review. But they understand the logic behind my approach.

Paradox #2: Clarity vs. Muddle

Up to this point, Deslauriers and Co. pursue a sensible path.

They know that MOST college profs use Method A (the bad one), so they want those profs to change. To encourage that change, the undertake a study showing a better option: Method B!

Given these research results, Deslauriers and Co. offer two clear and emphatic suggestions:

First: teachers should use Method B teaching strategies, not Method A strategies.

Second: to counteract students’ skepticism about Method B, we should explain the logic behind it.

What could be more helpful?

Alas, these clear suggestions can lead to another muddle. This muddle results from the freighted NAMES that this study gives to Methods A and B.

Method B — the good one — is called “active.”

Method A — the bad one — is called (inevitably) “passive.”

So, this study summarizes its findings by saying that “active” teaching is better than “passive” teaching.

These labels create real problems with the research conclusions.

Because these labels lack precision, I can apply them quite loosely to any teaching approach that I believe to be good or bad.

For instance: recall the experienced professor who regularly teaches this physics course. He mostly lectures; he also supplements the class with “demonstrations, … occasional interactive quizzes or conceptual questions.”

If I disapprove of that combination, I can call it “passive” — he mostly lectures!

If I approve, I can call it “active” — consider all those demonstractions, interactions, and conceptual questions!!

These labels, in other words are both loaded and vague — a perilous combination.

The peril arises here: literally no one in the world of cognitive science champions Method A.

EVERYONE who draws on cognitive science research — from the most ardent “constructivist” to the most passionate advocate for direct instruction — believes that students should actively participate in learning by problem solving, discussion, creation, and so forth.

Advocates for those two groups have different names for this mental activity: “desirable difficulties,” “productive struggle.” They think quite differently about the best way to achieve all that active thinking. But they all agree that students must struggle with some degree of difficulty.

Slippery Logic

This naming muddle creates unfortunate logical slips.

The study certainly suggests that Method B benefits students more than Method A. But, it doesn’t suggest that Method B is better than other methods that might reasonably be called by the open-ended named “active.”

For instance: it doesn’t necessarily mean that “constructivism” is better than direct instruction. And yet — because of those highly flexible labels — the study can be misinterpreted to support that claim.

My concern isn’t hypothetical. Someone sent me this study precisely to support the argument that inquiry learning promotes more learning than direct instruction.

But: “Method B” isn’t inquiry learning. And direct instruction isn’t Method A.

The Big Picture

I said at the beginning of this post that teachers might draw on research to be better teachers.

I worry that readers will draw this inaccurate conclusion based on this study:

“Research proves that ‘active learning’ (like projects and inquiry) is better than ‘passive learning’ (like direct instruction).”

Instead, this study suggests that asking students to do additional, productive mental work results in more learning than reducing their mental work.

Champions of both projects/inquiry and direct instruction want students to do additional, productive mental work.

Those schools of though have sharply different ideas of the best ways to accomplish that goal. But dismissing one of them as “passive” — and therefore obviously bad — obscures the important insights of that approach.


Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences116(39), 19251-19257.

A Smartphone Has, and Is, a Mirror Function
Andrew Watson
Andrew Watson

A recently published study looks at the role that ADULTS play in their children’s phone use.

In this post, I want to outline the study’s findings, and then consider their broader implications for schools and teachers.

The Study

In this recent study, led by Jason Nagata, researchers studied survey data from over 10,000 (yes, ten THOUSAND) adolescents. They looked for connections between “media parenting practices” and adolescent behavior.

So: if the family use cellphones at the dinner table, does that practice have an effect on — say — problems with school work.

Because we’re asking so many questions from so many people (10,000!), the researchers find A LOT of associations. (By the way, because these are survey data, we might be skeptical about the answers we get.)

Rather than go through each finding, I’ll highlight just a few:

  • An increase in cellphone usage at meals correlates with an increase in “problematic” social media behavior…
  • …and “problematic” video game behavior.
  • Ditto: an increase in cellphone usage in the child’s bedroom.
  • When parents reward or punish by allowing/restricting media usage, this practice correlates with increase in “problematic” video game usage.

This list could go on (and include dozens of caveats), but those headlines captured my attention.

In brief: if we’re worried about perils of technology usage, the cellphone might not be the best place to focus our efforts. Instead, the parents might be a good place to focus.

The Bigger Picture

Of course, this blog doesn’t focus on parenting. This study, instead, promted me to consider other questions:

Because the “should we ban cellphones in schools” debate runs so hot, it resists a nuanced conversation.

A baby wearing a suit with a "binkie" in his mouth, hoding and staring at a cellphone

I want to use this study to approach that debate from a slightly different angle.

As I read the case for banning cellphones, it typically includes the phrase “kids these days.” Sometimes directly, often indirectly, the argument implies that students themselves deserve the blame for their cellphone use.

I want to argue against that implication, for two reasons. (And then one more reason.)

First: for the most part, students do not buy themselves cellphones. Adults buy children cellphones. And we buy them the carrier plans and tchotchkes that go with them.

In other words, the public lament often sounds to me like this:

“How strange and terrible that children use the objects we bought them!”

If we don’t want our children to use an object, we don’t have to make them available.

(I understand that there’s a safety argument: “children need phones in case of emergency.” I also understand there are MANY ways to allow for emergency communication without smartphones. I’m looking at you, flip phone.)

This study described above helpfully highlights this point. If adults conjure up a problem, we should not turn around and criticize children for participating in it.

Resisting Temptation

The first reason not to blame children: adults probably bought them the smartphones.

The second reason not to blame children: everyone involved in the smartphone industry makes them as tempting as possible.

For instance: if I create video games for a living, I’m not bothered that children play my game during their classes — I’m DELIGHTED.

Presumably that kind of obsessive devotion means that the game will go, or has gone, viral.

Presumably I’ll get rich.

Heck, I might even reach the pinnacle of social success; I could become an influencer.

To expand on the point made above, the public lament often sounds to me like this:

“How strange and terrible that children use the objects that an  entire industry of experts has made as tempting and enjoyable as possible!”

If we don’t want our children to use a super fun object, we don’t have to make them available. *

And One More

I wrote above that we’ve got at least two reasons to spend less time blaming children for their cellphone use. I’d like to add one further reason.

(I should say: I am, at this moment, skating out on VERY thin ice. I hope you won’t be too upset with me.)

I’ve spent the last 12 years of my life leading PD workshops for teachers: workshops about cognitive science research, and the uses of that research in day-to-day teaching.

I have A LOT of anecdotal data about teachers in groups.

My biggest concern during these sessions is NOT “resistent behavior.” I almost never have grumpy teachers tell me to get off their lawn. Teachers typically react with great enthusiasm when they get to talk about a cog-sci approach to teaching.

My biggest challenge is: cellphones.

Many teachers simply won’t put them down.

Many school administrators simply won’t put them down.

Honestly: a PD talk sometimes feels like that screen-obsessed scene from WALL-E.

Here’s my point: it strikes me as arresting for teachers to be so angry at students for doing precisely what we’re doing.

To expand (once again) on the point made above, the public lament often sounds to me like this:

“How strange and terrible that students use their phones in the same way that we teachers do!”

If we don’t want our students to use smart phones, we should model the restraint we expect.

More important: the DIFFICULTY we have modeling that restraint holds up an unwelcome mirror. Every time I send a text during a faculty meeting, I should acknowledge my participation in the very cultural habit that I decry.

Solution, Please?

You might reasonably ask me: “if everything you’ve written is true, what should my school do about smartphones?”

That’s an entirely reasonable question. I have a tentative answer, and a confident answer.

Tentatively:

I see almost no upside to studets’ having and using phones during class; I see LOTS of downsides.

For that reason, I think that in most cases, schools should do everything they reasonably can to limit or prevent cellphones from being out during class.

I suspect — but don’t know — that most schools would benefit from a substantial ban on cellphone use during the day on campus. Students will talk with each other more if they’re texting each other less.

Confidently:

I think a school’s cellphone policy should include teachers’/adults’ cellphone use as well.

Of course: a school’s relationship with a teacher differs from its relationship with a student. But: teachers’ cellphone usage can absolutely undermine our authority to insist on students’ abstinence.

The phone is a mirror. It both communicates with others, and shows us more about ourselves. We ought to look into that discomforting mirror.


* Important note: I made this entire argument without using the word “addicted.” As I’ve written elsewhere, we cannot currently claim that “cellphones are addictive.” That’s a lively research-based debate, not a settled conclusion.


Nagata, J. M., Paul, A., Yen, F., Smith-Russack, Z., Shao, I. Y., Al-Shoaibi, A. A., … & Baker, F. C. (2024). Associations between media parenting practices and early adolescent screen use. Pediatric Research, 1-8.

Again with the Questions (Second of a Series)
Andrew Watson
Andrew Watson

Three weeks ago, I started a short series of blog posts about asking questions.

After all, we’ve got SO MUCH RESEARCH about questions, we need to keep track and make sense of it all.

To structure these posts, I’ve been focusing on these three topics:

When to ask a particular kind of question?

Who benefits most immediately from doing so?

What do we do with the answers?

So, for questions that we ask BEFORE learning starts (“before” is the “when”):

Teachers check our students’ prior knowledge for our own benefit; now we know how best to plan an upcoming unit.

and

We ask “prequestions” for students’ benefit; it turns out that — even though students don’t know the answers to prequestions — they benefit from trying to answer them.

So: here’s the next “when”…

DURING Class

We’ve explored the questions to ask BEFORE students start learning (prior knowledge, “prequestions”).

What about DURING the learning process?

Two students raising their hands and laughing as they look at each other

Here again, I think we’ve got two essential categories. Importantly, we should plan and think about these questions differently.

Firstchecking for understanding.

Of course, we want our students to understand the ideas and processes that we’re discussing in class today. Alas, “understanding” happens invisibly, inside our students’ minds.

The only way to ensure that they understand: ask them questions. Their answers will make that invisible understanding visible — or, perhaps, audible.

When checking for understanding, we should keep some key principles in mind:

We should check for understanding frequently throughout a lesson. The correct number of times will vary depending on context. As a high school teacher, I rarely go more than seven or eight minutes without some kind of check.

As Doug Lemov says: “reject self report.” Our students don’t understand today’s topic well enough to know whether or not they understand — so it NEVER helps to ask students “got it?”

Be sure that everyone answers the checking-for-understanding questions. Whether we use mini-whiteboards or cold calling or quizlet, we want as broad a sample as possible of our students before we move on to the next step of the topic.

We should ask tricky questions, but not trick questions. That is: the questions should be difficult enough to ensure that students genuinely understand the topic (that’s “tricky”), but we’re not trying to fool them (“trick”).

Of course, wise thinkers have LOTS more to say about checking for understanding, but these few principles give us a strong start.

Important Distinctions

So, “who benefits” from checking for understanding? And: “what do we do with the answers”?

Roughly speaking, the teacher benefits from checking for understanding. If I C4U and discover that my students DO understand, I know:

a) that my teaching method for those several minutes worked as I had hoped, and

b) that I can continue the lesson.

If my students DON’T understand, I know:

a) it didn’t, and

b) I shouldn’t.

In other words: checking for understanding provides useful feedback to the teacher.

What should I do with the answers to these questions?

The right and wrong answers I see/hear will guide me as I decide what to do next.

If, for instance, my students answer a question about direct objects incorrectly, I might not provide the correct answer right away. But I will draw on that feedback when I think about revising my lesson plan for the next 10 minutes.

During Class, Part 2

Of course, not all in-class questions focus on checking for understanding new material. I might — in fact, I should — devote some class time to reviewing and consolidating ideas that students have already learned.

To meet this goal, I will almost certainly rely on retrieval practice.

This blog has written EXTENSIVELY about retrieval practice, so I won’t do a deep dive here. You can check out previous posts, or savor this awesome website.

The headline is: students learn more not by reviewing material but by actively trying to retrieve it.

Rather than say: “Remember, the Ideal Gas Law says that pv=nrt.”

I should say

“Okay, please try to remember the Ideal Gas Law. Don’t should out; I’ll ask you to write it on your mini-whiteboards in a moment.”

We’ve got heaps o’ research showing that the extra mental effort required by retrieval helps consolidate memories.

Notice; I’m NOT trying to see if students have an initial understanding. When I taught this concept last week, I checked for understanding. My students DID understand it.

Instead, I’m trying to consolidate the understanding they had back then.

Important Distinctions

Once again: “who benefits” from retrieval practice? And: “what do we do with the answers”?

Whereas I, the teacher, benefit from  checking for understanding, MY STUDENTS benefit from retrieval practice. That mental effort helps them consolidate and transfer the ideas they retrieved.

(Of course, I do get useful feedback about the stickiness of their prior learning, but that’s not the primary goal of RP.)

What should I do with the answers? Especially wrong answers?

This question leads to a surprisingly intricate answer. The short version goes like this:

If I have time, it’s helpful to correct wrong answers to retrieval practice questions ASAP.

I should do so ESPECIALLY if the question touches on an important core idea or procedure.

But:

Students get the benefit of retrieval practice even if they get the answer wrong. As long as they come across the correct answer eventually, they’ll benefit.

This topic gets nuanced quickly, but the headline is: wrong answers aren’t tragedies in retrieval-practice world.

To Sum Up

We ask students questions BEFORE learning; we take stock of their prior knowledge, and seed future learning with prequestions.

DURING class, we frequently check for understanding to ensure that current learning is happening as we hoped.

And we ask retrieval practice questions about ideas and procedures learned before, in order to help them consolidate that learning.

If we understand the differences among the purpose for and response to these questions, we will use them more effectively.

The Great Exam Debate: A Conversation with a Colleague
Andrew Watson
Andrew Watson

A colleague recently reached out to ask me about research into the benefits and perils of final exams in high school. Her question inspired a fun exploration of research on the topic; I thought it would be helpful to share both our conclusions and our process with you.

A hand holding a pencil filling out answers on a stardardized test

Before I dig into our discussion, it might be helpful to pause and ask yourself this question: “do I already have a firm opinion about final exams?”

That is: do you believe that exams provide much-needed accountability and a chance for meaningful accomplishment? Do you believe they subtract valuable instruction time and add needless academic stress? Your prior beliefs will shape the way you read the upcoming post, so you’ll probably learn more if you recognize your own preconceptions.

With this first step in place, let’s explore…

Not as Easy as it Looks

Our conversation started with a frank admission: it would be INCREDIBLY difficult to investigate this question directly.

To do so, we’d need to teach two identical courses – one of which does have a final exam, and the other of which does not.

This proposal, however, quickly becomes impossible.

Let’s say that one section of my English class has a cumulative final exam, and the other has a cumulative final project. The differences between an exam and a project require all sorts of other changes to the course…so the two experiences wouldn’t be similar enough to compare as apples and apples.

Almost any other attempt to answer questions about final exams directly leads to similar problems.

This realization might discourage those of us who regularly turn to research. At the same time, it forces us to rethink our question quite usefully.

Instead of asking:

“Are final exams good or bad?”

We can ask:

“When we think about a year-long learning experience, how can we conclude those months most helpfully? What set of cognitive experiences consolidates learning most effectively? And: how does the answer to that question depend on the specific context of my school?”

With that framework in mind, let’s get started…

Old Friends

Longtime readers know that I rely on several websites to launch my research journeys. In this case, my colleague and I started at elicit.org. I put in this question:

“Do final exams in high school benefit or harm learning?”

The results from this search highlight the complexity of the question.

This paper by Khanna et al argues that cumulative final exams benefit students more than non-cumulative exams; these benefits appear both in the short term – immediately after the exam – and up to 18 months later. (Technically speaking, 18 months is a LONG TIME.)

When I checked out that study on my two other go-to websites (connectedpapers.com, scite.ai), I found other papers that, roughly speaking, arrived at the same conclusion. Strikingly, those other papers suggested that cumulative exams especially benefit either struggling students, or students with less prior knowledge.

On the other hand, back at my elicit.org search, this study by Holme et al produces this bleak conclusion:

“High school exit exams have produced few expected benefits and been associated with costs for disadvantaged students.”

A quick search on connectedpapers.com finds that – sure enough – other researchers have reached roughly similar conclusions.

“Promising Principles”

As expected, our review of existing research shows the difficulty of answering this final-exam question directly.

So, let’s try a different strategy: returning to our reformulated question:

“When we think about a year-long learning experience, how can we conclude those months most helpfully? What set of cognitive experiences consolidates learning most effectively?”

Two Learning and the Brain stalwarts – David Daniel and John Almarode – often invite teachers to think about cognitive science not as rules to obey (“do this”), but as “promising principles” that guide our work (“think about this, then decide what to do”).

So: do we have any promising principles that might guide our question about final exams? Indeed we do!

This blog has written about spacing, interleaving, and retrieval practice so often that there’s no need to rehash those ideas in this post. And, it’s easy to see how to apply these promising principles to cumulative final exams. After all, such exams…

… almost REQUIRE spacing,

… almost REQUIRE interleaving,

… create MANY opportunities for retrieval practice.

Of course, almost anything can be done badly – and preparation for final exams is no exception. But – done well – final exams invite exactly the kind of desirable difficulty that cognitive science champions.

Slam Dunk?

Perhaps, then, we have answered my colleague’s question: schools should—no, schools MUST—use cumulative final exams to enact cognitive science principles.

…Insert sound of record scratch…

That statement overlooks the second part of the revised question above:

“What set of cognitive experiences consolidates learning most effectively? And: how does the answer to that question depend on the specific context of my school?”

In this case, my colleague works at a school that champions a progressive educational philosophy. In other words, final exams sound like a terrible idea.

Her school has long favored cumulative capstone projects. And even a cursory discussion makes it clear that such projects – like cumulative final exams – invite spacing, interleaving, and retrieval.

(Yes, yes: capstone projects can be designed very badly. So can final exams. Both can also be done well.)

As long as those capstone projects deliberately and thoughtfully enact all those promising desirably difficult principles, they too can consolidate a year’s worth of learning.

TL; DR

My collegial conversation suggests that cognitive science research neither forbids nor requires final exams.

Instead, that research gives us ways to think about the summary work that we do with students. We can adapt these promising principles to align with our own school philosophy. That conceptual combination – more than a specific research study – will guide us most wisely.