Monthly Archives: June 2024

A few years ago, I visited an English Department meeting at a well-known high school. The topic under discussion: a recently published “English Department Guide to Annotation.”

After the meeting, one of the authors asked me what I thought of the guide and the research behind it.

I asked: “Well, what IS the research behind it?”

The teacher answered: “Oh, ALL the research says this.”

Although this answer (“ALL the research shows…”) is quite common, it always makes me nervous.

In the first place, it sounds like a dodge, doesn’t it? It’s the sort of answer I might give if I didn’t actually know much about the research.

In the second place: it’s never true. Because teaching and psychology and research are all so complicated, researchers NEVER get the precisely same answer when they study interesting, complex, and important questions about teaching.

Even retrieval practice — one of the most research-supported teaching strategies we have! — doesn’t have unanimous support in the research literature.

So: what does research say about annotating texts? I recently stumbled across a study that explores this question…

A Promising Start

A recently published study looks quite specifically at the benefits of teaching annotation. In this study, 125 8th grade students in a social studies class learned a specific method for annotation; another 125 served as the control group.

The students who learned this new method got LOTS of practice: at least 100 minutes over 6 weeks. The study used a “business as usual” control group, which means that the teachers just taught as they usually do for the students who didn’t learn about annotation.

The result: impressive!

In brief, the students who annotated thought the method was relatively easy to use. And, they scored higher on a reading-comprehension test.

If you speak stats, you’ll be impressed to see that the Cohen’s d was 0.46: an attention-getting number for a 6-week intervention.

So far, this study gives us reason to focus on teaching annotation.

Subsequent Concerns

And yet, I do have concerns: some specific to this study, and some more generally about applying research to classrooms.

SPECIFIC concern:

As you can see in the image above, the annotating method includes highlighting. Alas, researchers have looked at highlighting specifically, and found it to be largely unhelpful.

For instance, when John Dunlosky et al. evaluated the efficacy of various study strategies, they didn’t find much to recommend highlighting:

On the basis of the available evidence, we rate highlighting and underlining as having low utility. In most situations that have been examined and with most participants, highlighting does little to boost performance.

It may help when students have the knowledge needed to highlight more effectively, or when texts are difficult, but it may actually hurt performance on higher level tasks that require inference making. (Emphasis added.)

If this annotating method includes a widely-discredited strategy, I worry about the research behind it.

GENERAL concerns

As I’ve written before, I want to offer “research based teaching advice” only if LOTS of research supports the advice. You know your school, curriculum, and students better than I do — so I need SUBSTANTIAL reason to say “do this, not that.”

Alas, we have precious little research into the question of annotation.

I’ve used my go-to resources (scite.ai, connectedpapers.com, elicit.org), and found almost no research pointing one way or the other. (The study described above does mention other experiments … but as far as I can discover none of them focuses precisely on annotation.)

So, yes: we have ONE study saying that this annotation method helped 8th graders learn more in a social studies class. But I don’t think we should be very sure of that narrow finding…much less confident about extrapolating.

That is: do these results tell us anything about annotating in a high-school English class? I don’t think so.

Niche-y, But Important

A final concern merits a brief discussion here.

As noted above, the study uses a “business as usual” control group. That is: some students got A SHINY NEW THING (for 100 minutes!). And some students got … nothing special.

As you can imagine, we might easily conclude that the SHINY NOVELTY — not the annotation specifics — helped the students.

The study would have been more persuasive if the control group had learned a different reading comprehension strategy instead of annotation. In this case, we would have more confidence about the benefits (or lack of benefits) of annotation.

Generally speaking: when a study compares Something to Something Else, not Something to Nothing, it gives us greater reason to rely on its findings.

In Brief

When our students read, we want them to think about the text they’re reading.

We want them to …

… learn the ideas in the text,

… associate it with their prior knowledege,

… update their schema on the topic,

… consider weaknesses, questions, and omissions,

and so forth.

Is annotating the best way to ensure they do so? I’m not yet persuaded.

If you know of good research on the topic, I hope you’ll let me know!

Lloyd, Z. T., Kim, D., Cox, J. T., Doepker, G. M., & Downey, S. E. (2022). Using the annotating strategy to improve students’ academic achievement in social studies. Journal of Research in Innovative Teaching & Learning, 15(2), 218-231.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public interest, 14(1), 4-58.

The summer stretches before you like a beach of relaxing joy. With a guilty-pleasure novel in one hand and an umbrella drink in the other, how should you best plan for the upcoming school year?

Let’s be honest:

You might want to give yourself a break. School is STRESSFUL. Some down time with your best friends — perhaps a refreshing walk in the woods — getting back into a fitness routine … all these sound like excellent ideas to me.

If, however, you’re the sort of person who reads education blogs in the summer, well, you might be looking for some ideas on refreshing your teaching life.

Since you asked…

The Essential Specifics Within the Big Picture

The good news about research-based teaching advice?

We have LOTS and LOTS of helpful suggestions!

The bad news about research-based teaching advice?

Well: we have LOTS and LOTS of helpful suggestions!! Probably too many suggestions to keep track of.

If only someone would organize all those suggestions into a handy checklist, then you might strategically choose just a few of those topics that merit your attention. If this approach sounds appealing to you, I’ve got even more good news:

You can check out Sherrington and Caviglioli’s EXCELLENT book Walkthrus. This book digests substantial research into dozens of specific classroom topics (how to value and maintain silence; how to create a “no opt out” culture). It then offers 5-step strategies to put each one into practice.

In a similar vein, Teaching and Learning Illuminated, by Busch, Watson*, and Bogatchek, captures all sorts of teaching advice in handy visuals. Each one repays close study — in the same way you might closely study a Walkthru.

With these books, you can do a deep dive into as many — or as few — topics as you choose.

School Policy

The hot topics in education policy world are a) cell phones and b) AI.

As everyone knows, Jonathan Haidt’s recent book has made a strong case for heavily restricting cell phone usage for children.

I think it’s equally important to know that LOTS of wise people worry that Haidt is misinterpreting complex data.

Schools and teachers no doubt benefit from reading up on this debate. My own view, however, is that we should focus on the effects that phones (and other kinds of technology) have in our own schools and classrooms. Create policies based on the realities you see in front of you — not abtract data about people who might (but might not) resemble your students.

As for Artificial Intelligence: I think the field is too new — and evolving too rapidly — for anyone to have a broadly useful take on the topic.

In my brief experience, AI-generated results are too often flukily wrong for me to rely on them in my own work. (Every word of this blog is written by me; it’s a 100% AI-free zone.)

Even worse: the mistakes that AI makes are often quite plausible — so you need to be a topic expert to see through them.

My wise friend Maya Bialik — one-time blogger on this site, and founder of QuestionWell AI — knows MUCH more about AI than I do. She recommends this resource list, curated by Eric Curts, for teachers who want to be in the know.

A Pod for You

I’m more a reader than a pod-er, but:

If you’re in the mood for lively podcasts, I have two recommendations:

First, the Learning Scientists routinely do an EXCELLENT job translating cognitive science reseach for classroom teachers.

Unsurprisingly, their wise podcast is still going strong after many years.

Second, Dr. Zach Groshell’s podcast — Progressively Incorrect — explores instructional coaching, math and reading instruction, current debates in education, and other essential topics.

You might start with his interview with fan favorite Dan Willingham.

(Full disclosure: I have appeared on both podcasts, and am friends with the people who run them.)

The Journey Ends at Its Beginning

But, seriously, give yourself a break. You’ve worked hard. Take the summer off. I bet you’ve got A LOT of shows to binge-watch in your queue…

* A different “Watson”: EDWARD Watson. As far as I know, we’re not related.

In recent years, our field has seen a great increase in ‘one-pagers’: handy summaries of research topics and suggestions — complete with headings and friendly graphics.

Given their rising popularity, I thought it would be helpful to offer guidance on using them well.

So: what strategies will help you (and through you, your students) benefit the most from one pagers?

Strategy #1: Make Your Own

No, seriously.

The person who learns the most from a one-pager is the person who makes the one pager.

For instance: to make a 1-pager on retrieval practice, you first need to …

… learn a lot about retrieval practice;

… organize all your findings, preferably with lively and helpful graphics;

… understand its boundary conditions and best uses;

… summarize your findings in clear, consistent, understandable language;

… decide what information to include, and what to leave out…

In other words, you need to do all those generative learning activities that are most likely to ensure that you learn and remember new ideas. (Click here for a brisk review of the Ensers’ excellent book on generative learning.)

So, my most emphatic recommendation for 1-pagers: make ’em.

Strategy #2: Edit a 1-Pager

Let’s say I find a 1-pager on taking good classroom notes.

Because I know this research pool fairly well, I can both read the 1-pager and have a conversation with it.

That is: I can comment on the passages where I agree (“I’m so glad someone else has found this study helpful!”) and also note those passages where I don’t (“No, the Mueller and Oppenheimer study really does not support that claim.”).

I could re-prioritize its suggestions, reword a sentence or two, even sketch in a quick graphic that better encapsulates the ideas.

In effect, this editing strategy simply riffs on the first one above (“Make Your Own 1-Pager”). By collaborating or arguing with another 1-pager, I’m getting many of the same cognitive benefits.

Notice one key point here:

To succeed at strategies 1 and 2, I need to know a fair amount about the topic.

That is, relatively speaking I’m an expert.

If I were a novice on the topics of retrieval practice and note-taking, I couldn’t really create or respond to a 1-pager very substantively.

Strategy #3: Use a Single 1-Pager Intensively Over Time

A few recent books have focused on 1-pager summaries of cognitive science in the classroom.

For instance, consider Walkthrus by Tom Sherrington and Oliver Caviglioli. This book chooses several dozen specific classroom topics — cold calling, seating charts — and offers precise 5-step guidelines for examining, honing, and reviewing our practice.

Each of these Walkthrus is, in effect, a 1-pager; each one invites us to spend several days steeping in its advice.

So, if you choose the Walkthru on … say … “scaffolding,” you’ll start by reviewing all five steps. Then you’ll focus on step one: “map out the components of a task”.

This step will take at least a day or two, and during those 48 hours you’ll spend a lot of time thinking about that step.

So too with step 2 (“provide supports at a detailed level”), and then again with step 3 (“provide supports at an overview level”).

As you can easily see, this strategy means you’ll spend a lot of time — days and days! — thinking about the information, ideas, and suggestions on this particular 1-pager. The result: you’ll learn a lot about this topic.

Notice the key here: focused work over time. We don’t get the benefit of walkthrus by reading Walkthrus. We get the benefit by choosing one of them and dwelling on it for meaningful periods of time.

Less Productive 1-pager Strategies

I’m offering the list above because I worry that the most common use of a 1-pager is also  the least effective use.

Here’s the commom, ineffective use:

Step 1: I don’t know much about (for example) worked examples.

Step 2: I come across a 1-pager on worked examples.

Step 3: I read it, and think “wow, that makes so much sense!”

Step 4: I go on about my day.

This process creates the illusion of understanding; I believe that I have learned something about worked examples.

Alas, as we well know, learning almost never happens this way. I don’t learn about worked examples by reading a 1-pager about them, but by grappling substantively with the concept over time.

In other words, I fear a 1-pager paradox:

They DO benefit the people who create them (strategy #1);

They DO benefit people who already know a great deal about a topic (strategy #2);

They DO benefit novices who devote days to intensive study and practice (strategy #3);

But they DON’T benefit beginners who use them as their form invites us to use them: a quick read.

If teachers wants to learn about any meaningful educational topic (working memory, entry routines, growth mindset, turn-n-talk, spacing and interleaving) we can’t do so simply by reading a 1-pager. We have to read a book, attend a workshop, take a class…and then think and practice and fail and start over.

We know that students learn by this interative process of engaging, grappling, struggling, and transforming. Teachers learn this way too.

Adam Boxer (on ex/Twitter @AdamBoxer1) is one of the few people who also worries about this trend in public. Here’s a brief recent thread.