Yearly Archives: 2023

Here at Learning and the Brain, we like research-informed teaching suggestions.

At the same time, we remember Prof. Dan Willingham’s timeless motto: “one study is just one study, folks.”

That is: one study might show a particular conclusion – but one study isn’t entirely persuasive.

Instead, we’d like SEVERAL studies looking at roughly the same question. If those studies all point the same direction, then we can feel increasingly confident that this conclusion has merit.

For instance:

Several years ago I blogged about an impressive study by Dr. Jay Greene. It suggested …

… that attendance at live theater improved students’ knowledge of the play (not a big surprise)

… and, it improved students’ tolerance (surprise!)

… and, it enhanced their desire see the world from another’s perspective.

Because I REALLY WANT those statements to be true, I looked at this research with extra skepticism. But I was ultimately persuaded that the study design made lots of sense.

Of course, “one study is just one study.”

It would be GREAT to see more research on this topic. Well, I have good news…

Take Two

I’ve recently come across a second study looking at the benefits of live theater — this one led by Angela Watson (no relation that I know of).*

As is often the case, this research team looked at some of the same questions, and some additional questions as well.

So, they asked:

Does live theater provide benefit for students? How about art museums? How about symphonies?

Do these cultural field trips benefit students in different demographic groups?

The Greene study looked at tolerance and “social perspective taking” – the desire to see the world from another’s perspective. Watson’s research also asked questions about conscientiousness and empathy.

So, what did they find?

Methods and Conclusions

As you can imagine, these questions require lots of logistical complexity.

We can study – say – “retrieval practice” fairly easily in a classroom. But, schlepping large groups of students to plays, museums, and symphonies takes a lot more coordination, planning, and cooperation.

Basically, these researchers had students attend different numbers of plays, symphonies, and museums: as few as one trip, or as many as six.

And, they had students fill out questionnaires. And, they followed up for two years!

So, do additional “cultural” field trips make a difference for these students?

As is so often the case, the answer is both “yes” and “no.”

So, for instance, attending the theater/symphony/museum did NOT make students more interested in going back.

And, contrary to Greene’s study, it did NOT make students more “empathetic.”

Specifically, students responded to statements like these: “After seeing a play or a movie, I have felt as though I were one of the characters.”

Their responses did not vary depending on the number of field trips they went on.

At the same time, Team Watson did find some benefits.

Attending field trips did increase “social perspective taking.”

That is: students were asked questions like “When you are angry at someone, how often do you try to ‘put yourself in his or her shoes’?”

Students who attended more field trips put on those shoes more often.

Also, female students — although not males — demonstrated higher levels of conscientiousness after participating in more cultural field trips. (Alas, this change didn’t last as long as the others.)

The news isn’t all good, but not all bad either.

What About Tolerance?

The case of “tolerance” is especially tricky.

When Watson & Co. ran the numbers, these field trips didn’t affect the students’ answers to questions like:

“Some people have views you oppose very strongly. Do you agree that these people should be allowed to come to your school and give a speech?”

So, strictly speaking, field trips didn’t increase tolerance.

However, Watson speculates that the wording of these questions was especially complicated. And, she notes that the students in this cohort had relatively low reading scores.

So, perhaps, students really did feel more tolerant, but the question’s complexity masked that change.

Watson supports this hypothesis by noticing that the students at the high end of reading comprehension DID have higher tolerance scores.

In other words: the students who understood the question better expressed higher levels of tolerance — and might reflect a truer understanding of the field trips’ effects.

TL;DR

We now have even more research showing some SEL benefits for field trips to cultural events.

This trips…

… consistently help students “put themselves in someone else’s shoes,”

… might increase students’ tolerance of others’ opinions,

… enhance girls’ conscientiousness (albeit temporarily).

We have less certain evidence that field trips enhance empathy, or that they encourage students to attend more cultural events.

Equally important: having MORE studies on one topic allows us to contemplate these SEL benefits with greater nuance and sophistication.

* To be precise, Watson’s study is a “working paper.” It hasn’t been peer reviewed in an academic journal.

Watson, A., Greene, J., Holmes Erickson, H., & Beck, M. I. (2019). Altered attitudes and actions: Social-emotional effects of multiple arts field trips.

Because we teachers are a busy lot, we sometimes want simplicity and clarity:

“I’m honestly too busy to sort through all the options and variables; just tell me what to do.“

In fact, when I went to my first Learning and the Brain conference in 2008, that was exactly my plan.

The researchers would tell me what to do.

I would do it.

VOILA: brain-based teaching.

The more time I spend in this field, the more I doubt that logical chain. In fact, I frequently warn people against that kind of thinking.

My regular mantra:

“Don’t just DO this thing. Instead, THINK this way.”

In other words, we teachers always have to translate research-based advice to our own context.

Today’s News

I recently came across a study looking at twelve different instructional activities in English and math classrooms.

In this study, “instructional activities” include …

… open discussion among teacher and students,

… use of whiteboard by teacher

… students working in groups

… one-to-one teaching

… students copying from the whiteboard

And so forth.

The research team asked: do students benefit from the same instructional activites in both disciplines?

To answer this question, researchers had observers keep track of teachers’ instructional activities. These observers — during more than 2500 visits! —  recorded whether teachers did these 12 activities “none or very little,” “some of the time,” or “most or all of the time.”

And, they then looked at the students’ scores on national exams in English and math. (This study was done in England, where most students take the GSCE when they’re 16 years old.)

So, what did they find when they put all those pieces together.

Fascinating Results

First, this research team found that teachers do different things:

Some teachers spend much of class time using traditional direct instruction, including lecturing and the use of textbooks, while other teachers devote more class time to students working with their classmates or individual practice.

For instance: one third of teachers use “open discussion” most or all of the time, but one quarter don’t do so at all.

Second, those different instructional activities matter.

In math classes, students benefit from a) practicing on their own, and b) teachers’ checking for understanding.

Students who engage in these activities “all or most of the time” score significantly higher than those who do so “some of the time.” (In this case, “significantly higher” is a bit hard to describe. Probably the easiest way to say this is: both statisticians and the students themselves would notice the difference.)

In English classes, however, students benefit from working and talking with each other (and the teacher).

So, to answer the question in this post’s title: at least according to this study, we shouldn’t teach all disciplines in the same way.

What This Finding DOES Mean

If you teach math or English to high-school students in England, I think you should give this study a careful look to guide your classroom practice.

That is: I’ve given an introduction — but the study includes A LOT more information that could be practically helpful to you.

Even more important:

If you don’t fit in that teaching category, this study means that research-based teaching advice always requires translation and adaptation.

Students benefit from different instructional activities in math and English. And, presumably, in other disciplines as well.

That is: you might go to a conference session that highlights the importance of mind-maps. (I’m picking this example at random.) That session shows research about its effectiveness in helping students learn.

However, this study clearly reminds us that we might need to adapt that advice to our own classrooms.

High-school English teachers might have students create mind-maps together; remember, students benefit from “working and talking together.”

High-school math teachers might have them create mind-maps solo; students benefit from “working on their own.”

More generally, this study might prompt you to ask some direct questions during that mind-map session. Was the research done with students in different grades? In different school or community cultures? Studying different topics? With diagnosed learning differences?

In other words: this specific research finding reminds us of a general lesson. We should be curious about and open to research-based suggestions. AND, we should check and be sure the research aligns with our teaching context before we make drastic changes.

What This Finding DOESN’T Mean

The flipside of my last point is: this research should encourage you to adapt your teaching practices only if your classrooms look like these classrooms.

Do you teach history? This research might not (or might) talk directly to you.

Do you teach second grade? Ditto.

Perhaps you teach in a different cultural context — say, Korea, Cairo, or Kansas.

Perhaps your school has a specific teaching philosophy (a Montessori school; a military academy) that rules out these approaches.

In other words: don’t just DO what this research tells you to do. THINK about your teaching practice with these ideas in mind — and see if they fit and make sense in your world.

Burgess, S. M., Rawal, S., & Taylor, E. S. (2022). Teachers’ use of class time and student achievement (No. w30686). National Bureau of Economic Research.

If we’re going to make “research-based” claims about education — and we are! — then we better have the research goods.

So, how can we at Learning and the Brain — and you as teachers and administrators — find and evaluate research?

For a few years now, I’ve relied on two great sources.

Scite.ai helps me see if a particular study is an outlier, or in alignment with other research findings.

Connectedpapers.com shows a spider web of research studies on any given topic.

They’re great, and you should check them both out. (I’m on those sites several times a week.)

But, we’ve heard A LOT about AI in recent weeks. Can we rely on newer websites to help us out?

I spent an hour or so on Monday playing around on two sites; here’s what I found.

The Inevitable ChatGPT

I won’t rehash all the Twitter agita on this subject, but gosh ChatGPT has gotten lots of attention. (Twitter tells me it’s either the savior of education, or the end of education, a deranged stalker of New York Times reporters, or a nothingburger.)

Can it help us explore research into education?

To answer this question, I asked Chat GPT to describe “schema theory.” Here are the first two paragraphs of its answer:

“Schema theory” is a cognitive psychology theory that describes how people understand, organize, and store information about the world. The theory suggests that people use mental frameworks or “schemas” to categorize information and simplify their understanding of the world around them.

A schema can be thought of as a mental structure that represents knowledge about a particular category of information, such as objects, events, or social interactions. For example, a person might have a schema for a “restaurant” that includes information about the typical layout, menu items, and social norms associated with dining out.

So far, so good. (The restaurant example of schema is quite common.)

I asked it to dig a bit deeper, and to identify problems or limitations with schema theory. Again, I got a solid set of answers. For instance, a limitation:

Overgeneralization: Schemas can sometimes lead to overgeneralization, causing people to perceive things in a stereotypical or biased manner. For example, people with a schema of a “criminal” may unconsciously associate criminal behavior with certain groups.

When I asked it for research supporting schema theory, its first three answers were several decades old, so I asked it for research from the last 20 years.

Of the three it listed, this one looked most promising:

Lee, J. J., & Anderson, N. (2015). The role of schema in the interpretation of ambiguous scenes. Memory & Cognition, 43(8), 1186-1197.

But, strangely, when I tried to find that study on Google Scholar, it didn’t seem to exist.

Stranger still, when I went to the website for the journal Memory and Cognition, it also doesn’t exist. (To be clear: the journal exists; the article doesn’t exist in it.)

You’re not going to believe this, but it seems that ChatGPT simply MADE UP a research study supporting schema theory.

The Chat Gets Awkward

I pointed out to ChatGPT that the source doesn’t exist, and it responded:

I apologize, it seems I made a mistake. The study by Lee and Anderson (2015) that I mentioned in my previous answer does not seem to exist. As a language model, I strive to provide accurate and up-to-date information, but sometimes I may make errors or misunderstand the information given to me. I appreciate your feedback and will take this into account in my future responses.

Okay, I’m not happy about this. But it did fess up to its mistake, and promise to do better.

So, the conversation continued:

Few debates rage hotter in education circles than that between educational progressives and educational traditionalists. (I’m emphasizing “educational” in these phrases, because they don’t necessarily align with political trad/prog divides. This blog doesn’t do politics.)

One recent summary — relying heavily on Dewey — describes the debate this way:

Educational traditionalists “argue that teachers should carefully select and sequence the best knowledge from their subject areas and then deliver it directly to the whole class, while maintaining order.”

Whereas

Educational progressives “argue that teachers should focus on facilitating individualised learning experiences in which pupils can explore their natural inclinations, thus nurturing their interests and developing general thinking skills.”

Of course, the debate gets MUCH more complicated than these summaries, but it’s handy to have a quick definition. According to this summary article cited above, roughly 2/3 of teachers side with one or the other of these two positions.

But: do we have research favoring one approach or the other?

Problems and Solutions

Because both educational philosophies encompass substantial sets of teaching ideas — everything from pedagogy to curriculum to motivation to metacognition — they resist efforts to evaluate them in their entirety.

We might run a study that shows … say … this “ed prog” unit motivates 6th graders more strongly than the analogous “ed trad” unit. However, such a study doesn’t necessarily mean that the philosophy itself works for all students, all subjects, all cultures, and all definitions of “works.”

So, what to do?

Education scholars Dr. Sam Sims and Dr. John Jerrim have found an ENORMOUS data set from Germany that just might make this possible.

It shows how much academic progress several thousand German students made over several years.

It measures their expressed motivation for studying German and math.

Heck, it even tracks their metacognative facility.

And,

It asks questions about their teachers’ expressed place on the ed prog/ed trad continuum.

It also asks questions about the teachers’ educational practices (PBL, lecture), to see if they align with those expressed beliefs.

WOW.

If we crunch the numbers just right, we should be able to answer several questions:

Do students learn more in one or the other of these approaches?

Does their motivation vary depending on them?

Is one approach or another better for more or less successful students?

So, what do Sims and Jerrim discover when they run their equations?

Not What I Was Expecting

The blog title promises something “weird.” So, here goes…

Because the ed prog/ed trad debate has an ethical valance to it, it often prompts dramatic calls:

“This way is the right way, so our profession really must do it right! After all, anything else is wrong.”

This study — weirdly — comes to an astonishingly bland conclusion.

That is:

Which philosophy helped students learn more?

Honestly, both worked equally well.

But wait: which one helped struggling learners more?

Meh. Both worked equally well.

But surely one fostered student motivation more than the other!

Well, kinda. Educationally traditional teaching practices resulted in slightly higher levels of motivation in German. But, not in math. And, not much even in German.

Um, metacognition?

Again, no big difference — although a slight benefit for team ed trad.

So, this ferocious debate we’ve been having for decades? Maybe we’ve been arguing about the wrong topics…

Trying to Process

Honestly, I’m still trying to wrap my head around this research. (By the way, I heard about it from Peps Mccrea. If you haven’t signed up for his “Research Snacks,” do so NOW. And, you can hear his webinar on motivation March 18th.)

Here’s a random grab-bag of my early thoughts:

First, I don’t doubt that LOTS of people will simply reject these findings. One easy way to do so: they’re published not in a peer reviewed journal, but as a working paper.

Debates about the relative merits of peer review rage in the scholarly community. Readers who typically reject working papers for their lack of formal peer review might simply set this one aside.

Second, if these two approaches basically work equally well, then we shouldn’t focus on choosing one or the other: we should focus on doing both as well as we possibly can.

My own view is that cognitive science will help us do so. For instance: no matter my educational philosophical stance, my students will benefit if I understand how working memory works.

Third, the finding about motivation seems especially surprising — or at least provocative.

Champions of educationally progressive education typically trumpet motivation as one of its obvious benefits. (Hence the contemptuous phrase “drill and kill” to mock direct instruction.)

This research suggests that educationally traditional methods produce slightly higher levels of motivation (in one of the two subjects they measured).

But wait: if students in ed trad classes felt more motivation but didn’t learn more, something very strange is going on.

Perhaps (dare I write it?), motivation doesn’t matter for learning? (How can that possibly be?)

Perhaps (dare I write it?), ed trad methods produce slightly more motivation but slightly less learning — and those two effects balance each other out? (All my ed trad friends just howled in rage.)

Or perhaps there’s some other obvious explanation I’m missing?

Currently, I’m stumped.

Next Steps

Me, I’m going to watch the responses to Sims and Jerrim’s working paper, and see what additional wisdom shakes out.

If you’ve got additional or alternative perspectives, I hope you’ll share them in the comments.