Monthly Archives: April 2023

Sometimes teachers hear about research that SUPPORTS our current beliefs and teaching practice.

Honestly, that experience feels great. “Look,” says my interval voice, “I’ve been doing it right all along.”

And sometimes, we hear about research that CONTRADICTS our beliefs and practice.

Honestly, that’s a punch to the gut. “Wait,” says that voice, “I’ve been wrong all this time?”

This discomfort often prompts us to use a handy rejoinder: “well, you can find research that proves anything…”

The not so subtle implication: “yes, this research says that — but this research doesn’t really matter because even absurd positions can find research backing…”

So, what should we do when a colleague rejects our research-based advice with this claim? Or, what should we do when we find ourselves saying it?

Step #0: Let’s Check

In the first place, I’m honestly not so sure that we can find reseach that says anything.

Let’s take a common piece of teaching advice: “teachers should shake hands with their students as they enter the classroom.”

Can we find research supporting, or contradicting, this claim? If “we can find research that says anything,” we certainly should be able to.

Well, so far I haven’t found any research examining this question.

As I’ve written before, Dr. Clay Cook found that “positive greetings” at the door produced specific benefits for specific students.

But his research doesn’t remotely suggest that all teachers should “shake-in” at all times. (For one thing: “positive greetings” don’t have to be handshakes.)

I just asked Elicit.org this handshake-at-the-door question. The closest answer I found is a study showing that female professors get higher ratings on the first day of class when they shake in, whereas male professors get lower ratings.

But again: that study neither confirms nor contradicts the larger claim about daily handshakes.

It seems that we can’t always “find research that proves anything.”

Step #1: Start Reading

But let’s agree that we can find research supporting lots of strange conclusions — or, at least, conclusions I disagree with. What should we do when that happens?

Imagine that a friend tells me: “chewing gum increases learning.”

When I ask him if he’s found research supporting that position, he grins broadly and says: “check this out.”

So, let’s check it out!

A cursory glance suggests that — yes — my friend has found research supporting his position, but it’s not terribly persuasive research.

It includes exactly 16 participants.

It’s published in a journal that focuses on engineering (not, say, memory, or learning).

Its method of measuring attention is … well … HIGHLY unscientific.

In other words, my friend found research supporting this claim; however, I didn’t need to look very hard to find reasons to doubt it.

That is: it doesn’t really matter if I can find research that “proves anything.” What matters is if I can find GOOD research supporting a particular claim.

Step #2: Get Curious

But, is GOOD research enough? If I find one well-done study, should I accept that chewing gum does promote memory?

When I got started in this field, I noticed that the scholars I admired most shared a surprising intellectual habit:

They shift to curiosity.

That is:

Person 1 says: “research shows that chewing gum improves learning!“

Person 2 says: “nope; research shows that chewing gum has no effect at all on learning.”

At this point, person 1 might say: “well, you can find research that shows anything. You’re obviously wrong. My research is correct.”

Or, person 1 might say: “wow, I’m curious that we have seen research that arrives at contradictory conclusions! Let’s explore…”

Over the years, I’ve come to rely on two sources when I feel curious and want to explore.

Scite.ai asks how often a particular study has been cited overall; how many times its findings have been confirmed; and how many times its findings have been contradicted.

Connectedpapers.com looks at the most frequently cited papers related to the topic, and creates a cool spiderweb diagram to show their connections.

Using these websites, person 1 and person 2 can plug in their studies, and see how many OTHER studies arrive at their conclusions.

That is: rather that rely on just ONE study, we can look at a WHOLE GROUP of studies to reach our conclusion.

When I use these tools to explore the chewing gum claim, as I’ve written before, I arrive at several conclusions:

First: researchers have done a surprising amount of work on this topic. (It seems like SUCH a niche-y question that I’m surprised folks have investigated it substantively.)

Second: even the quality research in this field (i.e., more than 16 participants) arrives at contradictory results.

This overview, noting that we can find clear evidence of both benefits and detriments, concludes that “the robustness of reported effects of gum chewing on cognition has to be questioned.”

So, at this point I don’t think we can claim we have a decisive, research-informed answer to this question.

In other words: the question is not “can we find research that proves anything?”; or even “can we find GOOD research that points in a clear direction?”; but “can we find SEVERAL studies all pointing in a clear direction — and more studies pointing this way than that way?”

Only if the answer to that last question is “yes” should we teachers start changing our practice because “research says so.”

TL;DR

Can we really find research that supports any claim about education?

First: no.

Second: we don’t want research, we want good research.

Third: we don’t just want good research, we want several good studies pointing roughly toward the same conclusion.

Until we have met these criteria, we can’t really say that a particular claim merits our attention and respect.

Cook, C. R., Fiat, A., Larson, M., Daikos, C., Slemrod, T., Holland, E. A., … & Renshaw, T. (2018). Positive greetings at the door: Evaluation of a low-cost, high-yield proactive classroom management strategy. Journal of Positive Behavior Interventions, 20(3), 149-159.

Wilson, J., Stadler, J., Schwartz, B., & Goff, D. (2009). Touching your students: The impact of a handshake on the first day of class. Journal of the Scholarship of Teaching and Learning, 108-117.

Here’s a little expert advice on nutrition:

Michael Phelps — the most decorated Olympic athelete in any sport ever — obviously had to take EXCELLENT care of his body. He thought A LOT about fitness and nutrition.

While he was training for the Olympics, he ate roughly 10,000 calories a day.

So: if I want to attain peak fitness, I too should eat 10,000 calories a day.

If it’s good enough for Olympic medals winners, it’s good enough for me.

Wait a minute. [insert sound of record scratch]

That’s terrible advice.

10,000 calories per day might have been a good idea for Phelps. However — physically speaking — he and I have very little in common.

During his Olympic career, Phelps was in his teens and 20s.  I’m 57.

He was in peak human physical condition. I am — well — in very average physical condition.

He (I assume) undertook ferociously vigorous physical exercise — and burned calories — most of the day. I spend much of my day sitting here writing blog posts.

Basing my nutritional plan on Phelps’s example just makes no sense.

Simply put: stories of extreme human performance fascinate us. Alas, they rarely produce useful models for everyday life — or for teaching.

Danger, Will Robinson

That last paragraph, sadly, creates real problems for popular science writers.

In my experience, their formula goes something like this:

“Here’s a fascinating story abouts something EXTRAORDINARY that happened.

Now that I’ve got your attention, notice this AMAZING X FACTOR in my story.

Here’s some wonky research roughly related to Amazing X.

You should enact Amazing X in your life, too.”

Whether the extraordinary story focuses on burning planes or impossible inventions or heroic feats, those stories — we’re asked to believe — all have something to tell us about improving our lives.

But if it’s true, as I wrote above, that “stories of extreme human performance rarely produce useful models for teaching,” then the narrative structure above invites — heck, demands — our skepticism.

Amazing X might benefit extraordinary folks in outlier conditions. But, by definition, few of us teach in outlier conditions. Amazing X just won’t help us much. It might, in fact, be a very bad idea in our classrooms. (10,000 calories, anyone?)

Don’t Start Here

You have, perhaps, heard the story of the Mann Gulch Fire. (If not, you should check it out. It’s an AMAZING story.)

In brief:

Back in 1949, a group of trained “smoke jumpers” battled a wildfire that was burning toward the Missouri river. The fire abruptly turned towards them, and they realized they were trapped … and likely doomed.

In an instant, the group’s leader — “Wag” Dodge — came up with an astonishing solution. He set his own fire, and then stepped into its”shadow”: the area that his fire had burned clear. The wildfire burned around him — but not over the area that his fire had scorched.

Sadly, none of his men followed him into the shadow. Two other men outran the fire; most died.

This story appears in more than one book I know. The message: we want our students to think the way Dodge thought. We want them to be creative thinkers, who can come up with novel solutions to important problems.

I agree with those goals. I want my students to be able to think for themselves, and think past the knowledge that I have.

However: Dodge’s example tells us exactly nothing about helping students develop that capacity.

Dodge was a highly experienced firejumper. And he was in immediate danger of his life.

Our students are not highly experienced in the topic we’re teaching them. (If they were, we wouldn’t need to be teaching them.) And — except in very rare circumstances — they don’t face immediate peril.

Dodge’s thought process, in other words, has almost nothing to do with our students’ thinking. Until they know as much as Dodge knew, and have roughly as much experience as he had, we should have no expectation that they can “think the way he thought.”

We shouldn’t use his example to inform our work — even if it’s a great story.

Familiar Problems

Another example, from another popular science book:

Dr. K reads X-rays for a living. He found that he got bored and tired as the day progressed. He worried — reasonably enough — that he was getting sloppy as the day progressed.

So, he installed a “walking desk” in his office. He walked at a moderate pace as he read the X-rays, and felt much more alert and perceptive.

Dr. K wondered: does this technique benefit others?

He ran a study, and — sure enough!! — found that Doctors Who Walked spotted suspicious masses more often that Doctors Who Sat.

Clearly, walking is good for thinking. Therefore, teachers should have students walk as they learn.

Please insert a second [record scratch] here.

Once again: a great story about experts doesn’t meaningfully apply to the work we do in schools.

Doctors who read X-rays are highly trained experts. They’ve been in school for roughly two decades.

And: reading X-rays is a perception task.

If walking helps highly trained experts stay alert enough to perceive patterns better, we can ask if walking helps students learn better.

But both the people involved (experts vs. novices) and the cognitive task (perceiving established patterns vs. learning new patterns) are meaningfully different.

We really need research looking at this question directly before we make strong recommendations.

Based on my the research I know — and my experience as a classroom teacher:

Yes: exercise is good for the body, and good for the brain.

Yes: physical activity before learning provides lots of benefits. (Link)

No: physical activity during learning hasn’t been studied much. (Link)

And: based on my classroom experience, walking my students around outside while trying to discuss Macbeth with them seems like a deeply bad idea.*

Dr. K’s treadmill might help him and his colleagues; I don’t think it does much of anything for teachers and students.

TL;DR

When reading popular science books that include teaching advice, be aware:

The stories about extraordinary people doing extraordinary things fascinate and compel us.

However:

Before we make changes to our teaching practice, we should see research that looks at students like ours studying a topic like ours.

If we don’t, we’ll end up doing the teaching equivalent of eating 10,000 caleries a day.

* Yes, of course, if students are studying something that is in fact outside, it makes sense to go outside and look at it.

For instance: when I taught Where The Crawdads Sing — a book that relies heavily on the symbolism of marshes and swamps — I took my class out to see the marshes on school property.

I’m not saying: never take students for a walk. I am saying: do so with a very specific pedagogical purpose in mind.