{"id":7091,"date":"2023-05-07T08:00:23","date_gmt":"2023-05-07T13:00:23","guid":{"rendered":"https:\/\/braindevs.net\/blog\/blog\/?p=7091"},"modified":"2023-05-07T05:14:08","modified_gmt":"2023-05-07T10:14:08","slug":"have-i-been-spectacularly-wrong-for-years-part-1","status":"publish","type":"post","link":"https:\/\/www.learningandthebrain.com\/blog\/have-i-been-spectacularly-wrong-for-years-part-1\/","title":{"rendered":"Have I Been Spectacularly Wrong for Years? Part 1"},"content":{"rendered":"

Over the years, I\u2019ve used this blog to make several persistent arguments.<\/p>\n

One of those arguments features in almost every post I write: context always matters<\/em>.<\/p>\n

That is: research might suggest that a particular classroom strategy works well. However, teachers should always consider the fit <\/strong>between that research-informed suggestion<\/em> and their own teaching world<\/em>.<\/p>\n

Recently I read an article suggesting that my advice — whether or not it\u2019s true — is almost certainly useless.<\/p>\n

In brief: this article argues that too much focus on context and nuance increases complexity, and thereby makes the advice unpersuasive and easy to ignore. Wow<\/em>.<\/p>\n

Naturally, I followed up with the article\u2019s author: Dr. Morgan Polikoff<\/strong>. This transcript of our conversation has been edited for brevity and clarity.<\/p>\n

\n

Andrew Watson<\/strong>:<\/p>\n

Thank you Dr. Polikoff for taking the time to talk with me this afternoon.<\/p>\n

I\u2019ve been following your work on Twitter for a while because I like the topics you write about, and I\u2019m always interested in your point of view.<\/p>\n

So I was surprised recently to read an article you wrote that I disagreed with quite strongly.<\/p>\n

I do want to get to the disagreement part, but before we get there, I wonder if you could tell me what it is that you do and why you do it.<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I\u2019m an associate professor of K-12 policy at the USC Rossier School of Education. And my research primarily focuses on standards assessment and accountability policy and curriculum policy. That\u2019s been my bread and butter over the last 13 years at USC.<\/p>\n

\"Headshot<\/a><\/p>\n

I think of myself as a very public facing academic. In addition to traditional publishing in peer review journals and getting grants, I\u2019m really interested in bringing my research, and other people\u2019s research, to bear on pressing policy topics — so that it is actually having an impact<\/em> on policy and practice.<\/p>\n

That was the spirit in which I wrote the piece for Larry Ferlazzo\u2019s blog.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Your Education Week article<\/a> includes many ideas that I agree with.<\/p>\n

For instance, you start by saying:<\/p>\n

\u201cWhen communicating with the public, researchers should have one <\/strong>message and stick to it.\u201d<\/p>\n

That idea makes lots of sense to me.<\/p>\n

However, the next two points that you make post launch our disagreement.<\/p>\n

The first is: when communicating with the public, researchers should be more prescriptive<\/strong> in the advice that they give:<\/p>\n

\u201cAcademics in general are loath to offer specific recommendations about what policymakers and practitioners should do\u2026 This is unfortunately a recipe for your work not to matter, because if you don\u2019t offer specific, actionable guidance, someone else (probably someone less informed than you) will do it.\u201d<\/p>\n

And second, while researchers should provide some context and nuance for that research, there\u2019s always the danger of providing too much context and nuance<\/strong>:<\/p>\n

\u201cResearchers tend to want to nuance and contextualize their work to an excessive degree. And, of course, it\u2019s true that every child is different and every school is different, so no one intervention is going to work in every context. But the reality is that research is never going to be sufficiently contextualized for every individual setting and what works \u2018on average\u2019 is, in most cases, going to work in individual settings, too.\u201d<\/p>\n

Have I summarized that correctly? Do you want to say more about those two points?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

No, I think that\u2019s right.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Okay. I\u2019ll tell you why those two points troubled me and then maybe we can figure out if we still disagree.<\/p>\n

In the work that I do \u2013 explaining psychology and neuroscience research to teachers \u2013 I find that most people boil research down into highly prescriptive<\/strong>, almost script-like<\/em> messages.<\/p>\n

These scripts oversimplify the research and make it \u201cone size fits all.\u201d<\/p>\n

So I am regularly encouraging people to be less prescriptive<\/em> and to make allowance for context<\/em> as much as possible.<\/p>\n

Teachers shouldn\u2019t listen to the script<\/em> that the speaker is providing, but instead to think about<\/em> teaching the way researchers understand cognition, and then adapt that message to their own work.<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I think part of our disagreement is an audience<\/strong> difference.<\/p>\n

You\u2019re thinking primarily about teachers, I\u2019m thinking primarily about policy makers at various levels from state leaders down to district folks.<\/p>\n

And the second difference: this article was about impact<\/strong>.<\/p>\n

How should researchers think about maximizing their impact on things that actually go on?<\/p>\n

Now that said, I think that a lot of what I wrote about \u201cbeing more prescriptive\u201d does apply to working with teachers.<\/p>\n

There are lots of instances in which I think we avoid being prescriptive because we think we\u2019re not supposed to, or because we\u2019re fetishizing local control, or because we think that the domain of the classroom is the teacher\u2019s, or because we think that every child is utterly unique.<\/p>\n

And some of those things I think are true, but I also think that if you really want to move practices, you have to move systems<\/em>. And if you want to move systems, you have to tell people what to do a little more than I think we do now.<\/p>\n

When I say \u201ctell people what to do,\u201d I don\u2019t mean command<\/em> them. I mean give them very specific advice about what you think is the best thing to do.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Can you give me an example where prescription is beneficial, at least at the policy level?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I gave the example on Twitter of high-quality instructional materials<\/em>. I believe \u2013 and I will say this to any State Department of Ed person or any school district leader:<\/p>\n

… that every teacher should have a high-quality core instructional material in every subject that they teach,<\/p>\n

… having those materials should be a minimum expectation,<\/p>\n

… for the most part, teachers should use those materials as the core of their instruction.<\/p>\n

So that\u2019s me being quite prescriptive.<\/p>\n

But I think if I were to instead say something less prescriptive: like,<\/p>\n

\u201cWell, we think that curriculum materials on average are effective, but we really don\u2019t have evidence on this, or evidence is stronger in math than it is in other subjects,\u201d<\/p>\n

That\u2019s just a recipe for irrelevance. The purpose of the piece is about relevance<\/em>.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Just to play that out a little:<\/p>\n

If it\u2019s true<\/strong> that the evidence for high-quality instructional materials is better in math than in English (or foreign language teaching, or soccer coaching), I would hesitate to give prescriptive advice to anyone who isn\u2019t teaching math because I don\u2019t have evidence to support it<\/em>.<\/p>\n

Do you share that hesitation, or you\u2019re all in on giving broadly prescriptive advice?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I get what you\u2019re saying.<\/p>\n

I also think that the reality of evidence in education is so weak that if people listen to the advice that you just gave, that no one could really give advice about anything.<\/p>\n

We just don\u2019t have high quality causal studies with convincing evidence and lots of replication on virtually any topic. That\u2019s certainly the case on curriculum, even in math, which is the place where we have the most evidence.<\/p>\n

The best I can say, based on the highest quality evidence, is that I think that there\u2019s convincing evidence that some curriculum materials are more effective than other curriculum materials.<\/p>\n

So yeah, you\u2019ve got to make a lot of leaps and you can appropriately caveat.<\/p>\n

I think that you shouldn\u2019t lie about what the strength of the evidence is. Nonetheless, people have to make decisions and they have to make decisions now<\/em>. There are children in schools right now<\/em> who don\u2019t have curriculum materials, and one of the reasons for that is squishy people don\u2019t want to tell school districts that they must provide children with quality curriculum materials.<\/p>\n

And that is harming<\/em> those children, I am certain of it \u2013 even though I don\u2019t think that the convincing causal evidence is there.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

So let\u2019s consider an example that you and I discussed in our email exchange.<\/p>\n

In cognitive science world, people have insisted on something called the 10-minute rule<\/strong>: \u201cpeople can\u2019t pay attention to something they\u2019re not intrinsically interested in for more than 10 minutes.\u201d<\/p>\n

So, if you\u2019re a classroom teacher, you should design your lesson plan in 10-minute chunks.<\/p>\n

Sadly, when we look at the research behind the \u201c10-minute rule,\u201d it\u2019s comically weak. [link<\/a>]\n

Would I be applying your way thinking correctly were I just to say,<\/p>\n

\u201cWell, the 10-minute rule doesn\u2019t have great evidence, but it might be the only attention rule we\u2019ve got. So I\u2019m going proclaim that rule loud and strong. My claim isn\u2019t a hundred percent true, but it\u2019s better for teachers to plan in 10-minute chunks than 30-minute chunks.\u201d<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

Well, you\u2019re the expert on that topic and I\u2019m not, so I can\u2019t evaluate the evidence. If you think that the evidence doesn\u2019t support the 10-minute rule, then I think that you should say loud and clear that the evidence doesn\u2019t support that: the teacher should not do that.<\/p>\n

On the other hand, some people, I guess, think that the evidence is good, and I think they should feel free to \u2026 they\u2019re trying to have impact by saying that<\/em>.<\/p>\n

The reality is: people who are wrong can use these strategies, just like people who are right can use these strategies, and there\u2019s not that much I can do about that.<\/p>\n

And I think lots of times, people who are wrong or people who are trying to sell you something actually use a strategy like this because they recognize that it\u2019s effective.<\/p>\n

On the other hand, people who might understand the literature and the evidence more are reluctant to use this kind of direct language.<\/p>\n

Therefore, in some ways, this reluctance facilitates the people who are wrong and their negative impact on what goes on in classrooms.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

To contradict a false claim, should I say, \u201cthey\u2019re wrong!\u201d or should I provide my evidence for what\u2019s right?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I\u2019m not an expert on how to refute wrongness.<\/p>\n

My understanding of a refutation approach is: you say what the misconception is, you provide direct evidence that it\u2019s wrong, and then you tell people what they should do instead.<\/p>\n

I think being direct is the way to do that; not saying, “Well, it\u2019s right under some circumstances, but it\u2019s not right under others.”<\/p>\n

No, you need to be equally as direct as the people who are giving teachers bad advice — advice that you say is bad.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Which is complicated if the correct advice is, in fact, highly nuanced and context dependent.<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

If the advice you think is correct really is very complicated and you need to give different advice to every individual teacher or in every individual setting, then yes, I think that\u2019s going to be a hard message to get across. That\u2019s just the reality of the situation.<\/p>\n

If the message gets so complicated that you can\u2019t explain it in a few sentences, or a one page concrete piece of guidance that you can give to a teacher and say, \u201cHere\u2019s what I think you should do\u201d \u2013 then I think it\u2019s going to be hard to get them to do what you want them to do, what you think is good.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

That\u2019s dismaying but intriguing, isn\u2019t it?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

Listen, it\u2019s harder to get people to do complicated things than it is to get them to do simple things, and it\u2019s harder to get people to understand complicated things than it\u2019s to get them to understand simple things. Those seem like obvious statements.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

To some degree, those obvious statements create a challenge in this sort of work that I do, because the truth is:<\/p>\n

… second graders aren\u2019t 12th<\/sup> graders, and<\/p>\n

… math teaching isn\u2019t history teaching, and isn\u2019t soccer coaching and<\/p>\n

… the cultural context in Massachusetts isn\u2019t the cultural context in Reykjav\u00edk, isn\u2019t the cultural context in Korea.<\/p>\n

So it\u2019s just true that the specifics of teaching advice will be highly context dependent for all of those things, and many others.<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I don\u2019t know if I agree with you.<\/p>\n

I definitely agree with you, for instance, that kids develop, and so second graders are different from high schoolers.<\/p>\n

But as for cultural differences, I don\u2019t talk about Iceland or Korea; I talk only about the United States. The more relevant question is: are kids in Massachusetts that different from kids in Vermont or that different from kids in Nebraska?<\/p>\n

There are certainly cultural differences. Obviously there are racial and ethnic differences, there might be language differences.<\/p>\n

But I strongly suspect \u2013 and I think the literature is pretty clear on this — that much more of the variation in those kinds of things is within<\/em> school than between<\/em> schools.<\/p>\n

And so the advice that you give about how to teach, I don\u2019t think it\u2019s going to be fundamentally different. Math or reading in Massachusetts is not fundamentally different from math or reading in Nebraska or Texas — at least in terms of the cultural dimensions you\u2019re talking about.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

I think within the States, the cultural differences would be more — for instance — teachers at a Montessori school and teachers at a Catholic school and teachers at a military academy.<\/p>\n

Although the surrounding culture itself is substantially the same, these school cultures are inviting a different way of thinking about teaching and learning, and the relationships between teachers and learners, which resemble — if they don\u2019t equal \u2013 cross-national differences like Massachusetts and Iceland.<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

I\u2019m not sure how important that point is, but that makes sense.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Okay. I guess to wrap things up: is there anything that you haven\u2019t had a chance to say that you\u2019d like to say?<\/p>\n

Morgan Polikoff<\/strong>:<\/p>\n

Sure.<\/p>\n

Both in this conversation and also in the writing, I\u2019m doing what I advise people to do: oversimplifying things.<\/p>\n

And also, I\u2019m pushing back at you to be provocative. That\u2019s intentional. That\u2019s a strategy that I\u2019m using.<\/p>\n

I think that provocation is useful to people. Oftentimes people are uncomfortable or unwilling to say things that are even as provocative as this discussion\u2026and this is frankly pretty banal.<\/p>\n

I\u2019m playing up disagreement because I think that doing so is provocative and engaging to people.<\/p>\n

Andrew Watson<\/strong>:<\/p>\n

Well, it\u2019s certainly been engaging to me \u2013 and I suspect it will be to our readers as well. Thank you for taking the time to chat today.<\/p>\n

\n

This post’s title asks if I’ve been wrong for years. I don’t (yet) have an answer to that question. But, here are my quick take-aways from this conversation:<\/p>\n

A: Dr. Polikoff and I definitely disagree!<\/p>\n

B: I don\u2019t think I changed his mind.<\/p>\n

C: I\u2019ll let you know in a later post whether or not he has changed mine…<\/p>\n","protected":false},"excerpt":{"rendered":"

Over the years, I\u2019ve used this blog to make several persistent arguments. One of those arguments features in almost every post I write: context always matters. That is: research might suggest that a particular classroom strategy works well. However, teachers should always consider the fit between that research-informed suggestion and their own teaching world. Recently […]<\/p>\n","protected":false},"author":18,"featured_media":7095,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-7091","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-lb-blog"],"_links":{"self":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/7091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/comments?post=7091"}],"version-history":[{"count":5,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/7091\/revisions"}],"predecessor-version":[{"id":7097,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/7091\/revisions\/7097"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/media\/7095"}],"wp:attachment":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/media?parent=7091"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/categories?post=7091"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/tags?post=7091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}