Over the years, I’ve used this blog to make several persistent arguments.
One of those arguments features in almost every post I write: context always matters.
That is: research might suggest that a particular classroom strategy works well. However, teachers should always consider the fit between that research-informed suggestion and their own teaching world.
Recently I read an article suggesting that my advice — whether or not it’s true — is almost certainly useless.
In brief: this article argues that too much focus on context and nuance increases complexity, and thereby makes the advice unpersuasive and easy to ignore. Wow.
Naturally, I followed up with the article’s author: Dr. Morgan Polikoff. This transcript of our conversation has been edited for brevity and clarity.
Andrew Watson:
Thank you Dr. Polikoff for taking the time to talk with me this afternoon.
I’ve been following your work on Twitter for a while because I like the topics you write about, and I’m always interested in your point of view.
So I was surprised recently to read an article you wrote that I disagreed with quite strongly.
I do want to get to the disagreement part, but before we get there, I wonder if you could tell me what it is that you do and why you do it.
Morgan Polikoff:
I’m an associate professor of K-12 policy at the USC Rossier School of Education. And my research primarily focuses on standards assessment and accountability policy and curriculum policy. That’s been my bread and butter over the last 13 years at USC.
I think of myself as a very public facing academic. In addition to traditional publishing in peer review journals and getting grants, I’m really interested in bringing my research, and other people’s research, to bear on pressing policy topics — so that it is actually having an impact on policy and practice.
That was the spirit in which I wrote the piece for Larry Ferlazzo’s blog.
Andrew Watson:
Your Education Week article includes many ideas that I agree with.
For instance, you start by saying:
“When communicating with the public, researchers should have one message and stick to it.”
That idea makes lots of sense to me.
However, the next two points that you make post launch our disagreement.
The first is: when communicating with the public, researchers should be more prescriptive in the advice that they give:
“Academics in general are loath to offer specific recommendations about what policymakers and practitioners should do… This is unfortunately a recipe for your work not to matter, because if you don’t offer specific, actionable guidance, someone else (probably someone less informed than you) will do it.”
And second, while researchers should provide some context and nuance for that research, there’s always the danger of providing too much context and nuance:
“Researchers tend to want to nuance and contextualize their work to an excessive degree. And, of course, it’s true that every child is different and every school is different, so no one intervention is going to work in every context. But the reality is that research is never going to be sufficiently contextualized for every individual setting and what works ‘on average’ is, in most cases, going to work in individual settings, too.”
Have I summarized that correctly? Do you want to say more about those two points?
Morgan Polikoff:
No, I think that’s right.
Andrew Watson:
Okay. I’ll tell you why those two points troubled me and then maybe we can figure out if we still disagree.
In the work that I do – explaining psychology and neuroscience research to teachers – I find that most people boil research down into highly prescriptive, almost script-like messages.
These scripts oversimplify the research and make it “one size fits all.”
So I am regularly encouraging people to be less prescriptive and to make allowance for context as much as possible.
Teachers shouldn’t listen to the script that the speaker is providing, but instead to think about teaching the way researchers understand cognition, and then adapt that message to their own work.
Morgan Polikoff:
I think part of our disagreement is an audience difference.
You’re thinking primarily about teachers, I’m thinking primarily about policy makers at various levels from state leaders down to district folks.
And the second difference: this article was about impact.
How should researchers think about maximizing their impact on things that actually go on?
Now that said, I think that a lot of what I wrote about “being more prescriptive” does apply to working with teachers.
There are lots of instances in which I think we avoid being prescriptive because we think we’re not supposed to, or because we’re fetishizing local control, or because we think that the domain of the classroom is the teacher’s, or because we think that every child is utterly unique.
And some of those things I think are true, but I also think that if you really want to move practices, you have to move systems. And if you want to move systems, you have to tell people what to do a little more than I think we do now.
When I say “tell people what to do,” I don’t mean command them. I mean give them very specific advice about what you think is the best thing to do.
Andrew Watson:
Can you give me an example where prescription is beneficial, at least at the policy level?
Morgan Polikoff:
I gave the example on Twitter of high-quality instructional materials. I believe – and I will say this to any State Department of Ed person or any school district leader:
… that every teacher should have a high-quality core instructional material in every subject that they teach,
… having those materials should be a minimum expectation,
… for the most part, teachers should use those materials as the core of their instruction.
So that’s me being quite prescriptive.
But I think if I were to instead say something less prescriptive: like,
“Well, we think that curriculum materials on average are effective, but we really don’t have evidence on this, or evidence is stronger in math than it is in other subjects,”
That’s just a recipe for irrelevance. The purpose of the piece is about relevance.
Andrew Watson:
Just to play that out a little:
If it’s true that the evidence for high-quality instructional materials is better in math than in English (or foreign language teaching, or soccer coaching), I would hesitate to give prescriptive advice to anyone who isn’t teaching math because I don’t have evidence to support it.
Do you share that hesitation, or you’re all in on giving broadly prescriptive advice?
Morgan Polikoff:
I get what you’re saying.
I also think that the reality of evidence in education is so weak that if people listen to the advice that you just gave, that no one could really give advice about anything.
We just don’t have high quality causal studies with convincing evidence and lots of replication on virtually any topic. That’s certainly the case on curriculum, even in math, which is the place where we have the most evidence.
The best I can say, based on the highest quality evidence, is that I think that there’s convincing evidence that some curriculum materials are more effective than other curriculum materials.
So yeah, you’ve got to make a lot of leaps and you can appropriately caveat.
I think that you shouldn’t lie about what the strength of the evidence is. Nonetheless, people have to make decisions and they have to make decisions now. There are children in schools right now who don’t have curriculum materials, and one of the reasons for that is squishy people don’t want to tell school districts that they must provide children with quality curriculum materials.
And that is harming those children, I am certain of it – even though I don’t think that the convincing causal evidence is there.
Andrew Watson:
So let’s consider an example that you and I discussed in our email exchange.
In cognitive science world, people have insisted on something called the 10-minute rule: “people can’t pay attention to something they’re not intrinsically interested in for more than 10 minutes.”
So, if you’re a classroom teacher, you should design your lesson plan in 10-minute chunks.
Sadly, when we look at the research behind the “10-minute rule,” it’s comically weak. [link]
Would I be applying your way thinking correctly were I just to say,
“Well, the 10-minute rule doesn’t have great evidence, but it might be the only attention rule we’ve got. So I’m going proclaim that rule loud and strong. My claim isn’t a hundred percent true, but it’s better for teachers to plan in 10-minute chunks than 30-minute chunks.”
Morgan Polikoff:
Well, you’re the expert on that topic and I’m not, so I can’t evaluate the evidence. If you think that the evidence doesn’t support the 10-minute rule, then I think that you should say loud and clear that the evidence doesn’t support that: the teacher should not do that.
On the other hand, some people, I guess, think that the evidence is good, and I think they should feel free to … they’re trying to have impact by saying that.
The reality is: people who are wrong can use these strategies, just like people who are right can use these strategies, and there’s not that much I can do about that.
And I think lots of times, people who are wrong or people who are trying to sell you something actually use a strategy like this because they recognize that it’s effective.
On the other hand, people who might understand the literature and the evidence more are reluctant to use this kind of direct language.
Therefore, in some ways, this reluctance facilitates the people who are wrong and their negative impact on what goes on in classrooms.
Andrew Watson:
To contradict a false claim, should I say, “they’re wrong!” or should I provide my evidence for what’s right?
Morgan Polikoff:
I’m not an expert on how to refute wrongness.
My understanding of a refutation approach is: you say what the misconception is, you provide direct evidence that it’s wrong, and then you tell people what they should do instead.
I think being direct is the way to do that; not saying, “Well, it’s right under some circumstances, but it’s not right under others.”
No, you need to be equally as direct as the people who are giving teachers bad advice — advice that you say is bad.
Andrew Watson:
Which is complicated if the correct advice is, in fact, highly nuanced and context dependent.
Morgan Polikoff:
If the advice you think is correct really is very complicated and you need to give different advice to every individual teacher or in every individual setting, then yes, I think that’s going to be a hard message to get across. That’s just the reality of the situation.
If the message gets so complicated that you can’t explain it in a few sentences, or a one page concrete piece of guidance that you can give to a teacher and say, “Here’s what I think you should do” – then I think it’s going to be hard to get them to do what you want them to do, what you think is good.
Andrew Watson:
That’s dismaying but intriguing, isn’t it?
Morgan Polikoff:
Listen, it’s harder to get people to do complicated things than it is to get them to do simple things, and it’s harder to get people to understand complicated things than it’s to get them to understand simple things. Those seem like obvious statements.
Andrew Watson:
To some degree, those obvious statements create a challenge in this sort of work that I do, because the truth is:
… second graders aren’t 12th graders, and
… math teaching isn’t history teaching, and isn’t soccer coaching and
… the cultural context in Massachusetts isn’t the cultural context in Reykjavík, isn’t the cultural context in Korea.
So it’s just true that the specifics of teaching advice will be highly context dependent for all of those things, and many others.
Morgan Polikoff:
I don’t know if I agree with you.
I definitely agree with you, for instance, that kids develop, and so second graders are different from high schoolers.
But as for cultural differences, I don’t talk about Iceland or Korea; I talk only about the United States. The more relevant question is: are kids in Massachusetts that different from kids in Vermont or that different from kids in Nebraska?
There are certainly cultural differences. Obviously there are racial and ethnic differences, there might be language differences.
But I strongly suspect – and I think the literature is pretty clear on this — that much more of the variation in those kinds of things is within school than between schools.
And so the advice that you give about how to teach, I don’t think it’s going to be fundamentally different. Math or reading in Massachusetts is not fundamentally different from math or reading in Nebraska or Texas — at least in terms of the cultural dimensions you’re talking about.
Andrew Watson:
I think within the States, the cultural differences would be more — for instance — teachers at a Montessori school and teachers at a Catholic school and teachers at a military academy.
Although the surrounding culture itself is substantially the same, these school cultures are inviting a different way of thinking about teaching and learning, and the relationships between teachers and learners, which resemble — if they don’t equal – cross-national differences like Massachusetts and Iceland.
Morgan Polikoff:
I’m not sure how important that point is, but that makes sense.
Andrew Watson:
Okay. I guess to wrap things up: is there anything that you haven’t had a chance to say that you’d like to say?
Morgan Polikoff:
Sure.
Both in this conversation and also in the writing, I’m doing what I advise people to do: oversimplifying things.
And also, I’m pushing back at you to be provocative. That’s intentional. That’s a strategy that I’m using.
I think that provocation is useful to people. Oftentimes people are uncomfortable or unwilling to say things that are even as provocative as this discussion…and this is frankly pretty banal.
I’m playing up disagreement because I think that doing so is provocative and engaging to people.
Andrew Watson:
Well, it’s certainly been engaging to me – and I suspect it will be to our readers as well. Thank you for taking the time to chat today.
This post’s title asks if I’ve been wrong for years. I don’t (yet) have an answer to that question. But, here are my quick take-aways from this conversation:
A: Dr. Polikoff and I definitely disagree!
B: I don’t think I changed his mind.
C: I’ll let you know in a later post whether or not he has changed mine…
I think I’m still on your side
Thank you!