Skip to main content
Will Chess Make Me Better at Baseball?
Andrew Watson
Andrew Watson

Imagine for a moment that I’m coaching college baseball.

A baseball batter makes contact with a ball, hitting it out over an empty, brightly lit baseball diamond

I’ve noticed that my players have lots of specific skills, but lack the ability to make strategic decisions from an above-the-playing-field perspective. How can I help them improve?

Suddenly I think: aha! I’ll have my boys of summer learn chess. What better way to train them up in an above-the-field view? What better way to improve strategic decision making with that view in mind?

After all, the mental skills that they learn playing chess will — no doubt — transfer to playing ball.


I don’t know if a coach has actually tried this experiment. But I do know that a similar thought process drives A LOT of inquiry in the world of cognitive science.

If I want my students to learn history — or science, or tuba — I could teach them history, or science, or tuba.

Or, perhaps, I could boost their underlying brain power in some general way that will ultimately transfer to their history learning.

That is: rather than train their historical knowledge and thinking, I could enhance the cognitive resources with which they do the historical thinking. VOILA! More learning.

In my analogy, I could enhance my baseball players’ strategic vision and thinking (by teaching them chess); they can use their chess-enhanced vision as they play baseball.

So many possibilities…

Where to Begin; How to Proceed

If I want to pursue this path, I have LOTS of “cognitive resources” to choose from. Should I train my students attention? Or, one of their executive functions?

A research team has recently tried this approach with “cognitive control”: “a set of processes critical for guiding thoughts, feelings, and actions in a flexible, goal-directed manner.”

For their research method to be persuasive, it should meet several criteria. It should

1. Include enough people to make its results credible:

If a study includes 20 or 30 people, the results might be interesting, but won’t be compelling.

2. Test its results in both the short term and the long term:

When I train my baseball players with chess, I want them to preserve their chess-enhanced vision for a long time. If they lose that vision as soon as they stop playing chess, then they haven’t really improved their cognive function in a meaningful way.

3. Test those results in meaningful areas:

When I train my first baseman in chess, I’m happy if he gets better at chess. But I really want him to get better at baseball.

To be a little bit technical, I’m glad if I see “near transfer”: that is, chess training helps my players get better at speed chess. But I care about “far transfer”: that is, chess training helps my players spot the best place to force an out during a tricky fielding play.

Better and Better

This research team — led by Nikolaus Steinbeis — included some extra steps as well. I’m DELIGHTED to see that this study…

4. Includes a plausible comparison:

Researchers often take a worrisome shortcut. They try out a Cool New Thing — say, a curriculum, or a pedagogical strategy. When students learn more, they say: “look, our Cool New Thing enhanced learning.”

But this claim disguises a logical flaw. The benefits could come from doing SOME New Thing, not THIS New Thing.

To overcome this logical flaw, researchers should have an “active control group.” That is: some participants try THIS new thing, while another group tries a PLAUSIBLE NEW ALTERNATIVE thing.

If one group benefits more than the other, we can say that — yup — the change came from the curriculum itself, and not just from the newness.

5. Psych + Neuro

Wouldn’t it be wonderful if reseachers BOTH checked out psychological measures (“do the students learn more?”) AND neurobiological measures (“do their brains physically change?”).

Well, I’m happy to report that Team Steinbeis did ALL OF THESE THINGS.

1: The study included 250+ children, age 6-13. That’s not a HUGE number, but it’s noticeably larger than most studies.

2: They tested participants at the end of the study, and again a YEAR later. In my experience, very few studies have that kind of time horizon.

3: They checked to see if their cognitive control game improved participants’ cognitive control (“near transfer”). AND, they also check if it helped their learning, mental health, decision making, creativity, and resilience in the face of COVID stressors (“far transfer”).

4: This study included an active control group. Half the students played a video game with instructions that focused on improving their cognitive control. The other half played the same video game with instructions that focused on improving their response time. That’s a plausible alternative, no?

5: Researchers scanned relevant brain regions — inferior frontal gyrus, cingulo-opercular and fronto-parietal networks — to see if the training changed structure or function. (Don’t worry: I have only a dim understanding of what those words mean, and I’ve been in this field since 2008.)

Results, PLEASE

I’ve gone into more detail than usual because I want you to see why I find this study helpful and persuasive. As far as I can tell, this team has done everything right.

If training cognitive control helps students, we should see meaningful differences in far transfer effects — and in brain structure or function — after a year. This study design will let us see that.

So, their results? Close to nothing.

  • Cognitive control training didn’t help students learn more, or make better decisions. (Far transfer)
  • It didn’t make them more creative. (Also far transfer)
  • It didn’t didn’t change relevant brain structures, or the function of those structures.

Now, the training did help students do better at tests of cognitive control — even after a year. But we don’t really care about cognitive control on its own — that’s simply near transfer. We care about cognitive control because it usually helps with learning, and creativity, and so forth.

This research tells us: untrained cognitive control might predict academic success. But increasing cognitive control with computer game training does not result in greater academic success — or greater anything else.

In the language of my starting anecdote: my players got better at chess, but they didn’t get better at baseball. And — as the coach — I care about baseball.

The Big Picture

The idea that we can “train our students’ brains” has a lot of intuitive appeal. Perhaps because the claim includes the word “brain,” it gets lots of hopeful attention. (Because it includes the potential for enormous profits, it has lots of economic appeal as well.)

I wanted to focus on this study because it does such a careful job of rebutting that claim: at least as long as “cognitive control” is the particular element we’re trying to train.

In the future, if someone brings a brain training program to your attention, consider this research example. If that someone’s research method doesn’t include all of the steps above, you might hesitate before you invest scarce time and money in this approach.

Better, instead, to focus on teaching history, science, tuba — and baseball.


Ganesan, K., Thompson, A., Smid, C. R., Cañigueral, R., Li, Y., Revill, G., … & Steinbeis, N. (2024). Cognitive control training with domain-general response inhibition does not change children’s brains or behavior. Nature neuroscience27(7), 1364-1375.

Read This Post with Your Right Brain First…
Andrew Watson
Andrew Watson

My Twitter feed is suddenly awash with one of those “how does your brain?” work tests. (I should say, “tests.”)

If you look at the picture and see an angel, you’re right-brained.

If you see a helicopter, you’re left-brained.

This “test” has several important flaws.

Flaw #1: it’s not a helicopter or an angel — it’s obviously a dog.

Flaw #2: left-brain/right-brain is one of those zombie myths that just keeps coming back, no matter how many times we kill it.

Of all the myths in this field, this one puzzles me the most. Let me try to unpack my confusion.

Not True: The Brain

At the most basic level, this brain myth suffers from the flaw that it lacks any meaningful basis in neurobiological truth. In the world of theories about the brain, that’s a big flaw.

We can’t in any meaningful way find people who “use more of the right brain,” or “rely on left-brain thinking.”

If you’d like a detailed explanation of the wrongness here, I recommend Urban Myths about Learning and Education by de Bruyckere, Kirschner, and Hulshof.

If you’d rather click a link, check out this study. In the mild language of research, it concludes:

Our data are not consistent with a whole-brain phenotype of greater “left-brained” or greater “right-brained” network strength across individuals.

Translation: “people and brains just don’t operate that way. No seriously. They just don’t.”

Yes, yes: a few mental functions typically take place more on one side than another.

A conceptual image of a brain, falsely suggesting that the left hemisphere is computational and the right hemisphere is artistic

Back in grad school, we learned that 95% of right-handed people rely more on the left side of the brain for some reading functions. But 95% =/= 100%. And [checks notes] left-handed people do exist.

In any case, this finding doesn’t support the LB/RB claim — which is that some people rely more on these synapses, and others rely on those synapses.

Honestly: at the basic level of “how we use our brains,” we’re all “whole brained.” *

Not True: The Mind

Okay, so maybe the LB/RB claim isn’t exactly about “the brain” and more about “the mind.”

That is: some folks are more analytical (“left-brained”) and others are more creative (“right-brained”).

This version of the myth doesn’t use the word “brain” literally. (“Who knows precisely where those mental functions happen in the brain? We were just joshing, kind of poetically.”)

It simply argues that people think differently — and we can tidily divide them into two groups.

In other words, this version simply repeats the “learning styles” argument. These theories say we can divide students into distinct groups (visual/auditory/kinesthetic; or,  creative/analytical; or, happy/grumpy/sleepy) and then teach them differently.

Of course, the LB/RB version of “learning styles” is no truer than the other versions; they all lack solid evidence to support them.

The Myers-Briggs Type Indicator sort of claims to measure this distinction (“thinking vs. feeling”). But here again, we just don’t have good evidence supporting this test. **

So, whether we’re talking about neuroscience or psychology, LB/RB ain’t true.

Beyond “True”

One of my favorite quotations is attributed to George Box:

All models are false; some models are useful.

In other words: psychologists can offer a good model for how — say — working memory works. That model is “useful” because it helps us teach better.

However, that model is a model. The staggering complexities of working memory itself defy reduction into a model.

So, maybe LB/RB isn’t true, but is useful?

Honestly, I just don’t see how it could be useful.

If the model were true (it’s not) and I could divide my students into left and right brained groups (I can’t), what would I then do differently?

Just maybe I could devise a “creative” lesson plan for one group and an “analytical” lesson plan for the other. (I’m not sure how, but I’m trying to make this work.)

Yet: doing so would be an enormous waste of time.

Neither group would learn any more than they would with the same lesson plan. And all that time I dumped into my dual planning can’t be used to create an effective lesson plan.

That sound you hear is George Box weeping.

TL;DR

Left-brain/right-brain claims are NEITHER true NOR useful.

Do not take teaching advice from people who make them.


* Yes, it’s true, some people have only one hemisphere. But that’s really rare, and not at all what the LB/RB myth rests upon.

** Some time ago, I tried quite earnestly to find evidence supporting the MBTI. To do so, I emailed the company that produces it asking for published research. They did not send me any research; they did, however, sign me up for their emails.


Nielsen, J. A., Zielinski, B. A., Ferguson, M. A., Lainhart, J. E., & Anderson, J. S. (2013). An evaluation of the left-brain vs. right-brain hypothesis with resting state functional connectivity magnetic resonance imaging. PloS one8(8), e71275.

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological science in the public interest9(3), 105-119.

You Should Not (or Should) Let Your Students Take Pictures of Slides
Andrew Watson
Andrew Watson

Back in October, I wrote a blog post about a surprise: it turns out that students REMEMBER STUFF BETTER when they take photos of lecture slides.

For several reasons — including common sense — I would have predicted the opposite. In fact, so did the researchers (led by Dr. Annie Ditta) who arrived at this conclusion.

But when Team Ditta ran their study and crunched their numbers, they found that slide photos improved students’ recall.

Woman holding up mobile phono to take photo of speaker and slides

Having written that pro-photo blog post, I was genuinely alarmed to see a tweet from Prof. Dan Willingham — one of the greats in this field. He describes taking photos as “a terrible way to take notes.”

And Dr. Willingham should know. He’s just written a book focusing on study strategies — including note-taking.

What’s going on here? Have I given you terrible advice?

It turns out: Professor Willingham’s advice derives from this study, published in 2021 by Wong and Lim.

My blog post came from the Ditta study, published in 2022.

How do we explain — and choose between — studies that ask the same question and arrive at entirely different answers?

Untangling the Knot

Step 1: don’t panic.

It might seem that contradictory results explode the field of psychology. If THIS study shows “yes” and THAT study shows “no,” then the whole enterprise looks foolish and broken.

But here’s the thing:

Psychology is complicated.

Teaching and learning are complicated.

PEOPLE are complicated.

When psychology researchers study people who are teaching and learning, they’re studying FANTASTICALLY complicated topics.

For that reason, psychology researchers regularly produce contradictory results. That’s just how they roll.

And for that reason, no one study answers a question for good. To quote Dr. Willingham once again: “One study is just one study, folks.”

We should look not for one study to answer a question definitively, but for clusters of studies to point in a consistent direction.

If 10 studies show YES, and 2 studies show NO, and 2 more show CONFUSION — well then, “yes” strikes me as a plausible conclusion. (At least for now.)

Start Here

How can we know if most researchers have arrived at Wong’s 2021 conclusion (“photos = bad”) or at Ditta’s 2022 conclusion (“photos = good”)?

Step 2: Get curious.

Replace advocacy (“I know for sure that photos are good/bad!”) with curiosity (“I wonder what I’ll find? This should be fun…”)

For my curiosity projects, I rely on three websites: scite.ai, connectedpapers.com, and elicit.org. *

They all have different approaches and yield different kinds of results. And, they all help answer the question: “do we yet have a cluster of studies that mostly point to the same conclusion?”

So, what did I find when I asked those resources about the Wong (“photes = bad”) study?

When I looked on connectedpapers.com … it identified exactly ZERO other studies that asked questions about taking photos of lecture slides.

When I asked elicit.org a question on the topic … it came up with nothing.

Scite.ai did identify one other study responding to Wong. Sure enough, it’s the Ditta study: “photos = good.”

So, unless I’m missing something, we just don’t have much research on this topic. We can’t know where a “cluster of studies” might point because we don’t have anything remotely like a cluster.

Getting Specific

We’ve got at least one more research avenue to pursue:

Step 3: explore the boundaries.

Let’s imagine for a minute that Wong did her study with 3rd graders, and found that photos = bad; and (still imagining), Ditta did her study with college students, and found that photos = good.

In that case, we could reasonably imagine that they got different results because they studied participants in different grades.

Or (more imagining) maybe Wong studied photos of slides during a music class, and Ditta studied photos during an art history class.

Here again we could make a reasonable guess: slide photos will help in some disciplines (art!) but not others (music).

Researchers call these “boundary conditions”: as in, “this finding applies to people within these boundaries, but not outside them.

Potential examples: a conclusion applies to …

… math class but not history class, or

… a Montessori school but not a military academy, or

… for dyslexic students, but not for neurotypical readers, or

… in Icelandic culture, but not Brazilian culture.

You get the idea.

When we look at Wong’s and Ditta’s studies, however, we find they’re very similar. Adults watch short-ish videos, and do (or don’t) take photos or notes.

The studies differ slightly — Wong looks at mind wandering as an important variable, for instance — but not enough to draw strong conclusions.

At this point, neither our online resources nor our exploration of boundary conditions gives us any reason to prefer one study to the other.

End at the Beginning

No matter how the journey goes up to this point, we always end with …

Step 4: Look to your experience, and your colleagues.

In other words: we teachers should be curious (step 2) and informed (step 3). And, we always ultimately rely on our own judgement.

In this case — in my view — we simply don’t have a good research consensus to push us strongly one way or another. So, relying on my experience, here’s the policy I would follow with my 10th grade English students:

You may take pictures of photos or complex diagrams — anything that would be hard to put into words.

However, if you can put the material into words, I’m going to ask you to do so.

Why?

Because the more time you spend processing the information, the likelier it is you will understand and remember it.

This policy would, of course, have nuances and exceptions. (E.g.: dysgraphic students shouldn’t have to write as much.)

I want to emphasize, however, that your policy needn’t resemble my policy.

If you teach different kinds of students, or teach in a photo-friendly discipline (art history!), or if your experience tells you something else…you should follow your own wisdom.

TL;DR

Should students take photos of slides as a way to remember the material?

At present, we have so little research on the topic that it really can’t answer that question — ESPECIALLY because the studies contradict one another.

Instead, we should rely on our research-informed judgement.


* As I’ve written elsewhere, I would not use ChatGPT for this kind of inquiry. In my first forays into that world, the website simply MADE UP citations. Ugh.


Ditta, A. S., Soares, J. S., & Storm, B. C. (2022). What happens to memory for lecture content when students take photos of the lecture slides?. Journal of Applied Research in Memory and Cognition.

Wong, S. S. H., & Lim, S. W. H. (2021). Take notes, not photos: Mind-wandering mediates the impact of note-taking strategies on video-recorded lecture learning performance. Journal of Experimental Psychology: Applied.

Warning: Misguided Neuroscience Ahead
Andrew Watson
Andrew Watson

I recently ran across a version* of this chart:

An (inaccurate) chart listing neurotransmitters: their effects and activities that enhance them

As you can see, this chart lists several neurotransmitters and makes recommendations based on their purported roles.

If you want to feel love, you should increase oxytocin. To do so, play with your dog.

If you want to feel more stable, you should boost serotonin. To do so, meditate, or go for a run.

And so forth.

On the one hand, this chart seems harmless enough. It recommends that we do quite sensible things — who can argue against “self-care,” or “hugging your children”? — and so can hardly provoke much controversy.

I, however, see at least two reasons to warn against it.

Willingham’s Razor

Most everyone has read Dan Willingham’s Why Don’t Students Like School?  (If you haven’t: RUN, don’t walk, to do so.)

Professor Willingham has also written a less well known book called When Can You Trust the Experts?, which offers lots of wise advice on seeing though bad “expert” advice.

One strategy he recommends:

Reread the “brain-based” teaching advice, and mentally subtract all the brainy words. If the advice makes good sense without them, why were they there in the first place? **

In the lists above, do we really need the names of the neurotransmitters for that advice to make sense?

To feel a sense of accomplishment, accomplish something.

If you want to feel better, eat chocolate.

To calm down, walk (or run) outdoors.

Who could object to these suggestions? Do we need multi-syllable words to embrace them?

I worry, in fact, that such charts create bad mental habits for teachers. Those habits sound like this:

If someone knows complicated neuro-terminology, then their teaching advice must be accurate. When a blogger uses the phrases “corpus callosum” and “research says,” therefore, I have to take their teaching advice.

No, you really DON’T have to take their advice. LOTS of people use the language of neuroscience to make their suggestions sounds more authoritative.

As I’ve written elsewhere, neuroscience rarely produces classroom-ready teaching advice.

PSYCHOLOGY gives teachers great ideas about memory and attention and learning and motivation.

A biological understanding of what’s happening during those mental functions (i.e., neuroscience) is fascinating, but doesn’t tell teachers what to do.

In brief: beware people who use neuro-lingo to advise you on practical, day-to-day stuff. Like, say, that chart about “happiness chemicals.”

When Simplification Leads to Oversimplification

My first concern: the chart misleadingly implies that neuroscientific terminology makes advice better.

My second concern: the chart wildly oversimplifies fantastically complicated brain realities.

For instance, this chart — like everything else on the interwebs — calls oxytocin “the love hormone.”

A smiley face with the word "oxytocin" as the smile

However, that moniker doesn’t remotely capture its complexity. As best I understand it (and my understanding is very tentative), oxytocin makes social interactions more intense — in both positive AND NEGATIVE directions.

So: when we add oxytocin, love burns brighter, hatred smoulders hotter, jealously rages more greenly.

To call it the “love hormone” is like saying “the weather is good.” Well, the weather can be good — but there are SO MANY OTHER OPTIONS.

The statement isn’t exactly wrong. But its limited representation of the truth makes it a particular kind of wrong.

So too the idea that dopamine is a “reward chemical.” Like oxytocin’s function, dopamine’s function includes such intricate nuance as to be difficult to describe in paragraphs — much less a handy catchphrase. ***

By the way: the most comprehensive and useful description of neurotransmitters I know comes in Robert Sapolsky’s book Behave. As you’ll see, they’re REALLY complicated. (You can meet professor Sapolsky at our conference in February.)

TL;DR

Yes, walking outside and hugging children and exercising are all good ideas for mental health.

No, we don’t need the names of neurotransmitters to make that advice persuasive.

We might worry about taking advice from people who imply that neuro-lingo does make it more persuasive.

And we can be confident that neurotransmitters are much, MUCH more complicated than such simplistic advice implies.


* I’ve made my own modified version of this chart. The point of this blog post is not to criticize the individuals who created the original, but to warn against the kind of thinking that produced it. “Name and shame” isn’t how we roll.

** I’m paraphrasing from memory. I’m on vacation, and the book is snug at home.

*** [Update on 12/30/22] I’ve just come across this study, which explores some of the contradictions and nuances in the function of serotonin as well.

When Analogies Go Wrong: The Benefits of Stress?
Andrew Watson
Andrew Watson

An amazing discovery becomes an inspiring analogy:

Researchers at BioSphere 2 noticed a bizarre series of events: their trees kept collapsing under their own weight.

Why on earth would trees collapse? It doesn’t happen outside the BioSphere; so why would it happen inside?

And then the researchers figured it out. The BioSphere doesn’t have wind.

Trees react to the stress of wind by growing stronger. If they don’t get that beneficial stress, they can’t stand up when they become adult trees.

And here’s the heart-warming bit: that’s true for humans too.

As we grow and develop, we need some modest, reasonable stresses in our lives. Those small stressors make our emotional “tree trunks” strong, so we can manage the greater stresses of adult life.

I really want to make an uplifting poster right now — don’t you?

First Things First

This story that I’ve told begins with science: “Researchers at the Biosphere…”

And so, when I read that story, I felt a small shudder of delight. I can use this story to explain to students — and parents, and teachers — the benefits of reasonable/modest stresses in their lives.

After all, it’s a GREAT story, and a great analogy.

Even better, I can share the research behind it. (That’s what I do for a living: share research with teachers, students, and parents.)

However, the website where I first read that story doesn’t link to any research.

Hmmm.

So, I started looking.

This trees-need-wind story (and its uplifting analogy) shows up frequently on the interwebs. In fact, I think I notice two waves — one around 2013, another around 2020.

But, exactly none of the articles included any scientific links — much less links supporting the claim.

Glimmers of Hope?

When I switched from Google to Google Scholar, I did find this brief report.

It appears in Science magazine — a highly reputable source — and includes this sentence:

The trunks and branches of large trees became brittle and prone to catastrophic and dangerous collapse.

So, have I found the scientific backing that this analogy was missing?

Alas, this sentence is but one part of a long catalogue of problems in BioSphere 2, as noted in that report:

Vines grew “exceptionally aggressive[ly].”

19 of 25 vertebrate species went extinct.

“All pollinators went extinct.”

CO2 levels, oxygen levels, temperature, and light exposure all went haywire.

And, NONE of these problems has much of anything to do with wind.

In fact, the word “wind” doesn’t appear in this brief article.

Simply put: as far as I can tell, the whole “wind makes trees stronger” story sounds great, but has no research backing — certainly not at Biosphere 2.

Some Conclusions

First: does wind help strengthen trees?

Maybe.

I’ve been reading about something called — believe it or not — “reaction wood.” You can read about it here.

Second: does manageable stress benefit people in the long run.

Sure.

Check out “Yerkes-Dodson.”

Third: should we use uplifting-but-false analogies to communicate important scientific truths?

As long as Learning and the Brain is here, heck no.

When Do We Trust the Experts? When They Don’t Trust Themselves!
Andrew Watson
Andrew Watson

Back in 2010, three scholars published a widely-discussed paper on “Power Poses.” The headlines: when people adopt a strong stance (say, fists on hips, like Superman), they…

…take more risks in gambling tasks,

…change various hormone levels, and

…answer questions more confidently in interviews.

In other words, simply changing the way we stand can affect meaningful variables in our biology, and our performance on life tasks.

A TED Talk on the subject has gotten more than 61 million views. (Yes: 61,000,000!)

Of course, any claim this provocative may generate controversy. Sure enough, skeptics weighed in with counter-claims.

Then, in 2016, something quite shocking happened: one of the original researchers publicly withdrew her support for the claim.

Researcher Dana Carney wrote, with bracing forthrightness, “I do not believe that power pose effects are real.” (As you can see in this link, Carney herself put those words in bold type.)

She went on to list her concerns about the initial study (small sample size, “flimsy” data, and so forth), to include her skepticism on her CV, and to discourage others from studying the topic. *

Wow!

What Next?

In theory, science is gradually “self-correcting.” That is: if one group of researchers arrives at an incorrect conclusion, other researchers will – over time – sleuth out their mistakes. (Max Plank wryly observed that the process might take a long time indeed. In his grim formula, opponents don’t change their minds; they die out.)

Looking at Carney’s example, researcher Julia Rohrer wondered if we could speed that process up. What would happen, she wondered, if we gave researchers a chance to change their minds? What if we invited them to do what Carney did?

She and her colleagues spread the word that they hoped researchers might publicly self-correct. As she puts it:

“The idea behind the initiative was to help normalize and destigmatize individual self-correction while, hopefully, also rewarding authors for exposing themselves in this way with a publication.”

The result? Several did.

And, the stories these thirteen researchers have to tell is fascinating.

In the first place, these self-corrections came from a remarkably broad range of fields in psychology. Some researchers studied extraversion; others, chess perception. One looked at the effect that German names have on professional career; another considered the credibility of Swedish plaintiffs.

One – I’m not inventing this topic – considered the relationship between testosterone and wearing make-up.

Stories to Tell

These researchers, in fact, went into great detail — often painful detail — during their self-corrections.

They worried about small sample sizes, overlooked confounds, and mistakes in methodology. They noted that some replications hadn’t succeeded. Several acknowledged different versions of “p-hacking”: a strategy for finding p values that hold up under scrutiny.

A few, in fact, were remarkably self-critical.

Tal Yarkoni wrote these amazing words:

I now think most of the conclusions drawn in this article were absurd on their face. … Beyond these methodological problems, I also now think the kinds of theoretical explanations I proposed in the article were ludicrous in their simplicity and naivete—so the results would have told us essentially nothing even if they were statistically sound.

OUCH.

With equally scathing self-criticism, Simine Vazire wrote:

I cherry-picked which results to report. This is basically p-hacking, but because most of my results were not statistically significant, I did not quite successfully p-hack by the strict definition. Still, I cherry-picked the results that made the contrast between self-accuracy and peer accuracy the most striking and that fit with the story about evaluativeness and observability. That story was created post hoc and chosen after I had seen the pattern of results.

Others, however, critiqued their own methodology, but held up hope that their conclusions might be correct; “These claims may be true, but not because of our experiment.”

What Should Teachers Do?

These self-corrections might tempt us, or our colleagues, to cynicism. “See? Science isn’t objective! Researchers are just makin’ stuff up…”

I would understand that reaction, but I think it misses the point.

In truth, all ways of knowing include weaknesses and flaws.

Science, unlike many ways of knowing, acknowledges that awkward truth. In fact, science tries to build into its methodology strategies to address that problem.

For this reason, research studies include so many (gruesomely tedious) details.

For this reason, psychology journals require peer review.

Indeed, for this reason, researchers try to replicate important findings.

Obviously, these strategies at self-correction don’t always work. Obviously, researchers do fool themselves…and us.

However, every time we read stories like these, they remind us that — as a profession — scientists take correction (and self-correction) unusually seriously.

In fact, I think the teaching profession might have something to learn from these brave examples.

How often do schools — how often do teachers — admit that a success we once claimed might not hold up under scrutiny?

As far as I know, we have few Yarkonis and Vizires in our ranks. (I certainly have never made this kind of public statement.)

In brief: this kind of self-correction makes me trust both the profession of psychology and these individual researchers even more. If you’re conspicuously willing to fess up when you’re wrong, you deserve a much stronger presumption of trustworthiness when you ultimately say you’re right.


* By the way: one of Carney’s co-authors continues to defend power poses emphatically. You can read Amy Cuddy’s response at the end of this article.

 

“Compared to What”: Is Retrieval Practice Really Better?
Andrew Watson
Andrew Watson

When teachers turn to brain research, we want to know: which way is better?

Are handwritten notes better than laptop notes?

Is cold-calling better than calling on students who raise their hands?

Is it better to spread practice out over time, or concentrate practice in intensive bursts?

For that reason, we’re excited to discover research that shows: plan A gets better results than plan B. Now we know what to do.

Right?

Better than What?

More often than not, research in this field compares two options: for instance, retrieval practice vs. rereading.

Often, research compares one option to nothing: starting class WITH learning objectives, or starting class WITHOUT learning objectives.

These studies can give us useful information. We might find that, say, brief exercise breaks help students concentrate during lectures.

However, they DON’T tell us what the best option is. Are exercise breaks more helpful than retrieval practice? How about video breaks? How about turn-n-talks?

When research compares two options, we get information only about the relative benefits of those two options.

For that reason, we’re really excited to find studies that compare more than two.

Enriching Encoding

A recent podcast* highlighted this point for me.

A 2018 study compared THREE different study strategies: rereading, enriched encoding, and retrieval practice.

Participants studied word pairs: say, “moon-galaxy.” Some of them studied by reviewing those pairs. Some studied with retrieval practice (“moon-__?__”).

Some studied with enriched encoding. This strategy urges students to connect new information to ideas already in long-term memory. In this case, they were asked, “what word do you associate with both “moon” and “galaxy”?

My answer to that question: “planet.” Whatever answer you came up with, you had to think about those two words and their associated ideas. You enriched your encoding.

Because this experiment looked at three different study strategies, it gives us richer insights into teaching and learning.

For instance, students who reviewed remembered 61% of the word pairs, whereas those who enriched their encoding remembered 75% (Cohen’s d = 0.72). Clearly, enriched encoding is better.

But wait, what about students who used retrieval practice?

Even Richer

Students in the retrieval practice group remembered 84% of their word pairs.

So, yes: “research shows” that enriched encoding is “better than review.” But it’s clearly not better than retrieval practice. **

In fact, this point may sound familiar if you read last week’s blog post about learning objectives. As that post summarized Dr. Faria Sana’s research:

Starting class with traditional learning objectives > starting class without traditional learning objectives

but

Starting class with learning objectives phrased as questions  > starting class with learning objectives phrased as statements

In fact, Sana looked at a fourth choice:

Teachers immediately answer the questions posed in the learning objectives >?< teachers don’t immediately answer the questions posed in the learning objectives.

It turns out: providing answers right away reduces students’ learning.

Because Sana studied so many different combinations, her research really gives us insight into our starting question: which way is better?

Friendly Reminders

No one study can answer all the questions we have. We ALWAYS put many studies together, looking for trends, patterns, exceptions, and gaps.

For instance, boundary conditions might limit the applicability of a study. Sana’s research took place in a college setting. Do her conclusions apply to 10th graders? 6th graders? 1st graders? We just don’t know (yet).

Or, if you teach in a school for children with a history of trauma, or in a school for students with learning differences, or in a culture with different expectations for teachers and students, those factors might shape the usefulness of this research.

By comparing multiple studies, and by looking for studies that compare more than two options, we can gradually uncover the most promising strategies to help our students learn.


* If you’re not following The Learning Scientists — their website, their blog, their podcast — I HIGHLY recommend them.

** To be clear: this study focuses on a further question: the participants’ “judgments of learning” as a result of those study practices. Those results are interesting and helpful, but not my primary interest here.

The 10-Minute Rule: Is The Lecture Dead?
Andrew Watson
Andrew Watson

The “10-minute rule” offers teachers practical guidance. It typically sounds something like this:

If students aren’t intrinsically interested in material, they can pay attention to it for no more than 10 minutes.

Ergo: teachers should do something different every ten minutes.

Ergo: the lecture is dead.

I first heard the “10-minute rule” at a conference in 2008, and run across it frequently when I work with teachers. They too, it seems, heard it at a conference.

Any rule that gets so much love at teaching conferences must be true, right?

Research-Aligned Teaching Advice

If you’re reading this blog, you want your teaching to have research behind it. So, what exactly is the research behind the “10-minute rule?”

Neil Bradbury is glad you asked. He looked into its history, and came up with some astonishing results: results that would be funny if they weren’t so alarming.

Let’s start with a Johnstone and Percival study from 1976, where two researchers visited 90 lecture classes (!). By comparing observations, they agreed that attention started to wane within the first five minutes (!!), with another decrease in the 10-18 minute range (!!!).

As Bradbury reports, however, this conclusion gets murky quickly:

First: they visited only 13% of those lectures together. In other words: 87% of their data come from one lone observer.

Second: they don’t report how they measured attention, or — for that matter — lapses in attention.

That student looking out the window: is she distracted by a bird, or concentrating on the professor’s complex argument?

That student looking keenly at the slides: is he engrossed in the topic, or trying to remember his lines for tonight’s rehearsal?

Johnstone and Percival have no way to know.

In other words: the “10-minute rule” rests on the hunchy sense of two visitors who were — as far as we can tell — simply relying on their guts. Whatever we call that, we don’t call it “research.”

And, whatever we do with their hunches, we shouldn’t change our teaching because of them.

Measuring Attention

This study highlights a complex problem. Attention, of course, takes place inside our heads. How can we measure it?

One solution: keep track of students’ note taking. Perhaps, students take more notes when they pay attention, and fewer notes when they don’t?

If that hypothesis is true, then students who write less are paying less attention. When we find a steep decline in note taking, we’ve found the moment when attention has dropped off. Sure enough: a 10-minute increment turns out to be crucial.

Alas, as Bradbury points out, this approach also collapses.

First: students take notes relatively consistently throughout a lecture. Their note taking falls off in the final ten minutes, not after the first ten minutes.

Second: in fact, the quantity of note taking results from the professor’s lecture, not from the point in the lecture. When the speaker makes key points, students write more. When the professor is recapping, or simply winding down — as she might do at the end of a lecture — they take fewer notes.

As Bradbury pithily summarizes this approach:

Note taking is not a good proxy for attention whatsoever, and even it if were, it does not support a 10- to 15- minute limit on student engagement.

BOOM.

Let’s Get Physical

If note-taking doesn’t measure attention, perhaps we can use biological measures instead.

Research by Bligh used a pulsemeter to measure students’ alertness. This study found that their pulses dropped roughly 14% over the course of the class.

At last: research confirmation of the “10-minute rule”?

Alas, Bligh’s research found the same results during a discussion class as during a lecture.

We might think that a lower pulse suggests less attention. If it does, then neither class format sustains attention.

Classroom Implications

In brief, the “10-minute rule” isn’t a rule, and doesn’t last ten minutes.

More precisely: we have no research suggesting it’s a rule with a consistent time limit.

Given that truth, what should we teachers do?

First: focus on the obvious truth that people are different.

Older students can (probably) pay attention longer than younger ones.

Hungry students (probably) pay less attention than less-hungry ones. (Except right after lunch.)

Some cultures prioritize focused attention more than others.

Some lecturers know how to hold an audience better than others.

Your approach to teaching should vary based on your specific circumstances, not be dictated by an arbitrary rule (which sounds authoritative but has no research backing.)

For instance: I’m currently teaching two sections of the same class — one in person and the other online. I teach them differently because attention can be more difficult online. (And because the online section meets earlier in the day — a real challenge for high school students.)

Second: study the science of attention.

Surprisingly, attention isn’t one thing.

Instead, attention is a behavioral combination of three distinct mental processes.

The more teachers understand that complex mix, the more successful we can be in creating the behavior by managing the mental processes.

I’ve written a book on this subject: Learning Begins: A Classroom Teacher’s Guide to Working Memory and Attention. (Ryan Reynolds will play me in the movie, I’m sure.)

Or, you can read LOTS of great articles: here’s one place to start.

Whichever approach you take, don’t let implausible absolute rules shape your thinking. Pay attention to your students, and to attention itself. Those two beacons will guide you on your classroom journey.


In the past, I’ve cited Wilson and Korn’s 2007 discussion of this topic. My thanks to Zach Groshell (Twitter handle: @MrZachG) for pointing to Bradbury’s wonderful article.

When Evidence Conflicts with Teachers’ Experience
Andrew Watson
Andrew Watson

Here’s an interesting question: do students — on average — benefit when they repeat a grade?

As you contemplate that question, you might notice the kind of evidence that you thought about.

Perhaps you thought: “I studied this question in graduate school. The research showed that answer is X.”

Perhaps you thought: “I knew a student who repeated a grade. Her experience showed that the answer is X.”

In other words: our teaching beliefs might rest on research, or on personal experience. Almost certainly, they draw on a complex blend of both research and experience.

So, here’s today’s question: what happens when I see research that directly contradicts my experience?

If I, for instance, think that cold calling is a bad idea, and research shows it’s a good idea, I might…

… change my beliefs and conclude it’s a good idea, or

… preserve my beliefs and insist it’s a bad idea. In this case, I might…

… generalize my doubts and conclude education research generally doesn’t have much merit. I might even…

… generalize those doubts even further and conclude that research in other fields (like medicine) can’t help me reach a wise decision.

If my very local doubts about cold-calling research spread beyond this narrow question, such a conflict could create ever-widening ripples of doubt.

Today’s Research

A research team in Germany, led by Eva Thomm, looked at this question, with a particular focus on teachers-in-training. These pre-service teachers, presumably, haven’t studied much research on learning, and so most of their beliefs come from personal experience.

What happens when research contradicts those beliefs?

Thomm ran an online study with 150+ teachers-in-training across Germany. (Some were undergraduates; others graduate students.)

Thomm’s team asked teachers to rate their beliefs on the effectiveness of having students repeat a year. The teachers then read research that contradicted (or, in half the cases, confirmed) those beliefs. What happened next?

Thomm’s results show an interesting mix of bad and good news:

Alas: teachers who read contradictory evidence tended to say that they doubted its accuracy.

Worse still: they started to rely less on scientific sources (research) and more on other sources (opinions of colleagues and students).

The Good News

First: teachers’ doubts did not generalize outside education. That is: however vexed they were to find research contradicting prior beliefs about repeating a year, they did not conclude that medical research couldn’t be trusted.

Secondteachers’ doubts did not generalize within education. That is: they might have doubted findings about repeating a year, but they didn’t necessarily reject research into cold calling.

Third: despite their expressed doubts, teachers did begin to change their minds. They simultaneously expressed skepticism about the research AND let it influence their thinking.

Simply put, this research could have discovered truly bleak belief trajectories. (“If you tell me that cold calling is bad, I’ll stop believing research about vitamin D!”) Thomm’s research did not see that pattern at work.

Caveats, Caveats

Dan Willingham says: “one study is just one study, folks.” Thomm’s research gives us interesting data, but it does not answer this question completely, once and for all. (No one study does. Research can’t do that.)

Two points jump out at me.

First, Thomm’s team worked with teachers in Germany. I don’t know if German society values research differently than other societies do. (Certainly US society has a conspicuously vexed relationship with research-based advice.) So, this research might not hold true in other countries or social belief systems.

Second, her participants initially “reported a positive view on the potency of research and indicated a higher appreciation of scientific than of non-scientific sources.” That is, she started with people who trusted in science and research. Among people who start more skeptical — perhaps in a society that’s more skeptical — these optimistic patterns might not repeat.

And a final note.

You might reasonably want to know: what’s the answer to the question? Does repeating a year help students?

The most honest answer is: I’m not an expert on that topic, and don’t really know.

The most comprehensive analysis I’ve seen, over at the Education Endowment Foundation, says: NO:

“Evidence suggests that, in the majority of cases, repeating a year is harmful to a student’s chances of academic success.” (And, they note, it costs A LOT.)

If you’ve got substantial contradictory evidence that can inform this question, I hope you’ll send it my way.

EduTwitter Can Be Great. No, Really…
Andrew Watson
Andrew Watson

Twitter has a terrible reputation, and EduTwitter isn’t an exception.

The misinformation.

The name-calling.

The “team” rivalries: all heat and little light.

Did I mention the misinformation?

You might wonder: why bother? Honestly, I wouldn’t blame you if you didn’t. I myself was hesitant to sign up.

Despite all these flaws — none of which is exaggerated, by the way — I do find lots of benefits. This experience recently got my attention.

The Setup

On my personal Twitter account, I posted a link to research that had me puzzled. According to a small study, the motor cortex does not “remap” to represent prosthetic limbs.

Given all the research we have into neuroplasticity, I was genuinely shocked by that finding.

In fact, I’m currently reading Barbara Tversky’s book Mind in Motion, which talks about brains remapping in response to TOOL USE.

If brains remap because of tools, but not because of prosthetics — which are, from one perspective, tools that have been attached to the body — well: that’s a very strange.

But, people on Twitter know things I don’t. I thought: maybe someone knows more about this research pool than I…

Rising Action

Soon after I posted that link, my Twitter friend Rob McEntarffer (@rmcenta) retweeted it, sharing my curiosity. (By the way: “Twitter friends” are really a thing. I know LOTS of people — too many to name here — whom I have come to respect and like entirely by “meeting” them on Twitter. I would NOT have predicted that.)

One of his Twitter followers — someone I have never met and don’t know — retweeted Rob’s retweet, with a question to her professor.

So, we’re now at 3 or 4 degrees of separation. What happens next?

The Payoff

Turns out: this professor — whom I also don’t know — has lots of expertise in this research field. He briskly explained why the study couldn’t draw strong conclusions. (If I understand him correctly, its measurement methodology doesn’t allow it to make those claims.)

In other words: within a few hours, I went from…

being ASTONISHED because a research finding dramatically contradicted my (fairly basic) understanding of neural remapping,

to…

having a SUCCINCT AND CLEAR EXPLANATION why that research shouldn’t concern me,

and…

feeling RELIEVED that my understanding of neuroplasticity wasn’t so wrongheaded.

And, what made those changes possible — or, at least, a whole lot easier? Twitter.

Caveats

To be clear, Twitter really does include (and produce) foul, cruel nonsense. If you look for that, you’ll find it. (Tom Lehrer says: “Life is like a sewer. What you get out of it depends [at least in part] on what you put into it.”)

At the same time, I routinely come across generous teachers & researchers. They freely share perspectives and resources and contacts and information.

If you can stand the background noise, you might give it a look.

One place to start: @LearningAndTheB. Perhaps I’ll see you there.