{"id":3205404,"date":"2024-12-15T08:00:43","date_gmt":"2024-12-15T13:00:43","guid":{"rendered":"https:\/\/www.learningandthebrain.com\/blog\/?p=7969"},"modified":"2025-05-06T12:12:34","modified_gmt":"2025-05-06T17:12:34","slug":"early-thoughts-on-a-i-research-in-schools","status":"publish","type":"post","link":"https:\/\/www.learningandthebrain.com\/blog\/early-thoughts-on-a-i-research-in-schools\/","title":{"rendered":"Early Thoughts on A.I. Research in Schools"},"content":{"rendered":"<p>I hope that one of my strengths as a blogger is: <em>I know what I don&#8217;t know<\/em> &#8212; and I don&#8217;t write about those topics.<\/p>\n<p>While I DO know a lot about cognitive science &#8212; working memory, self-determination theory, retrieval practice &#8212; I DON&#8217;T know a lot about technology. And: I&#8217;m only a few miles into my own A.I.\u00a0journey; no doubt there will be <em>thousands<\/em> of miles to go. (My <a href=\"https:\/\/www.learningandthebrain.com\/blog\/chatgpt-and-beyond-the-best-online-resources-for-evaluating-research-claims\/\" target=\"_blank\">first foray <\/a>along the ChatGPT path, back in February of this year, did <em>not go well<\/em>&#8230;)<\/p>\n<p><a href=\"https:\/\/www.learningandthebrain.com\/blog\/wp-content\/uploads\/2024\/12\/AdobeStock_986829441-1.jpeg\"><img decoding=\"async\" class=\"alignright size-medium wp-image-7974\" src=\"https:\/\/www.learningandthebrain.com\/blog\/wp-content\/uploads\/2024\/12\/AdobeStock_986829441-300x200.jpeg\" alt=\"A young child types on a laptop; a small robot points out answers on a see-through screen that hovers between them\" width=\"300\" height=\"200\" \/><\/a><\/p>\n<p>Recently I came across research that looks at A.I.&#8217;s potential benefits for <em>studying<\/em>. Because I know studying research quite well, I feel confident enough to describe this particular\u00a0experiment and consider its implications for our work.<\/p>\n<p>But before I describe that study&#8230;<\/p>\n<h2>Guiding Principles<\/h2>\n<p>Although I&#8217;m not a student of A.I., I AM a student of thinking. Few cognitive principles have proven more enduring than Dan Willingham&#8217;s immortal sentence: &#8220;memory is the residue of thought.&#8221;<\/p>\n<p>In other words, if teachers want students to\u00a0<strong>remember<\/strong> something, we must ensure that they\u00a0<strong>think<\/strong> about it.<\/p>\n<p>More specifically:<\/p>\n<ul>\n<li>they should think about it\u00a0<em>successfully<\/em> (so we don&#8217;t want to overload <strong>working memory<\/strong>)<\/li>\n<li>they should think about it\u00a0<em>many times<\/em> (so\u00a0<strong>spacing<\/strong> and\u00a0<strong>interleaving<\/strong> will be important cognitive principles<\/li>\n<li>they should <em>think hard <\/em>about it (so\u00a0<strong>desirable difficulty<\/strong> is a thing)<\/li>\n<\/ul>\n<p>And so forth.<\/p>\n<p>This core principle &#8212; &#8220;memory is the residue of thought&#8221; &#8212; prompts an obvious concern about A.I. in education.<\/p>\n<p>In theory, A.I. simplifies complex tasks. In other words, it reduces the amount of time I think about that complexity.<\/p>\n<p>If artificial intelligence\u00a0<em>reduces<\/em> the amount of time I that I&#8217;m required to think about doing the thing, it necessarily\u00a0<em>reduces<\/em> the amount of learning I&#8217;ll do about the thing.<\/p>\n<p>If &#8220;memory is the residue of thought,&#8221; then less thinking means less memory, and less learning&#8230;<\/p>\n<h2>Who Did What?<\/h2>\n<p>Although discussions of generative A.I. often sound impenetrable to me, <a href=\"https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=4895486\" target=\"_blank\">this study<\/a> followed a clear and sensible design.<\/p>\n<p>Researchers from the University of Pennslyvania worked with almost 1000 students at a high school in Turkey. (In this kind of research, 1000 is an unusually high number.)<\/p>\n<p>These students spent time REVIEWING math concepts they had already learned. This review happened in three phases:<\/p>\n<p style=\"padding-left: 30px;\">Phase 1: the teacher re-explained math concepts.<\/p>\n<p style=\"padding-left: 30px;\">Phase 2: the students practiced independently.<\/p>\n<p style=\"padding-left: 30px;\">Phase 3: the students took a test on those math concepts. (No book; no notes; nada.)<\/p>\n<p>For all students, phases 1 and 3 were identical. Phase 2, however, gave researchers a chance to explore their question.<\/p>\n<p style=\"padding-left: 30px;\">Some students (let&#8217;s call them <strong>Group A<\/strong>) practiced in the usual way: the textbook, their notes, paper and pencil.<\/p>\n<p style=\"padding-left: 30px;\"><strong>Group B<\/strong>, on the other hand, practiced with ChatGPT at hand. They could ask it questions to\u00a0assist with their review.<\/p>\n<p style=\"padding-left: 30px;\"><strong>Group C<\/strong> practiced with a specially designed ChatGPT tutor. This tutor was programed not to give answers to students&#8217; questions, but to provide <em>hints<\/em>. (There were other differences between the ChatGPT and the ChatGPT tutor, but this difference strikes me as most pertinent.)<\/p>\n<p>So: did ChatGPT help?<\/p>\n<p>Did the students in Groups B and C have greater success on the <em>practice problems<\/em>, compared to Group A?<\/p>\n<p>Did they do better on the <em>test<\/em>?<\/p>\n<h2>Intriguing Results<\/h2>\n<p>The students who used A.I. did better <em>on the practice problems<\/em>.<\/p>\n<p style=\"padding-left: 30px;\">Those who used ChatGPT scored 48% higher than their peers in Group A.<\/p>\n<p style=\"padding-left: 30px;\">Those who used the ChatGPT tutor scored (are you sitting down?) 127% higher than their peers in Group A.<\/p>\n<p>Numbers like these really get our attention!<\/p>\n<p>And yet&#8230;we&#8217;re more interested in knowing how they did on the <strong>test<\/strong>; that is, how well did they do <em>when they couldn&#8217;t look at their books<\/em>, or ask Chatty questions.<\/p>\n<p>In brief: had they LEARNED the math concepts?<\/p>\n<p style=\"padding-left: 30px;\">The students who used regular ChatGPT scored <em>17% lower<\/em> than their notes-n-textbook peers.<\/p>\n<p style=\"padding-left: 30px;\">Those who used the ChatGPT tutor scored <em>the same as<\/em> those peers.<\/p>\n<p>In brief:<\/p>\n<p style=\"padding-left: 30px;\">A.I. helped students succeed <em>during practice<\/em>.<\/p>\n<p style=\"padding-left: 30px;\">But, because it reduced the amount of time they had to THINK about the problems, it didn&#8217;t help them\u00a0<em>learn<\/em>.<\/p>\n<p>Case closed.<\/p>\n<h2>Case Closed?<\/h2>\n<p>In education, we all too easily rush to extremes. In this case, we might easily summarize this study in two sentences:<\/p>\n<blockquote><p>&#8220;A.I. certainly didn&#8217;t help students learn; in some cases it harmed their learning. Banish A.I.!&#8221;<\/p><\/blockquote>\n<p>While I understand that summary, I don&#8217;t think it captures the full message that this study gives us.<\/p>\n<p>Yes: <em>if we let students ask ChatGPT questions<\/em>, they think less and therefore learn less. (Why do they think less? Probably they simply ask for the answer to the question.)<\/p>\n<p>But:\u00a0<em>if we design a tutor that offers hints not answers<\/em>, we reduce that problem &#8230; and eliminate the difference in learning. (Yes: the reseachers have data showing that the students spent more time asking the tutor questions; presumably they had to <em>think harder<\/em>\u00a0while doing so.)<\/p>\n<p>As a non-expert in this field, I suspect that &#8212; sooner or later &#8212; wise people somewhere will be able to design A.I. tutors that are <em>better at asking thought-provoking hints<\/em>. That is: perhaps an A.I. tutor might cause students to think even MORE than other students praticing the old-fashioned way.<\/p>\n<p>That two sentence summary above might hold true today. But we&#8217;ve learned this year that A.I. evolves VERY rapidly. Who knows what next month will bring.<\/p>\n<h2>TL;DR<\/h2>\n<p>Although THIS study suggests that A.I. doesn&#8217;t help (and might harm) learning, it also suggests that more beneficial A.I. tutors might exist in the future.<\/p>\n<p>If &#8212; and this is the ESSENTIAL &#8220;if&#8221; &#8212; if A.I. can prompt students to THINK MORE than they currently do while practicing, then well-established cog-sci principles suggest that our students will learn more.<\/p>\n<hr \/>\n<p>* A note about the publication status of this study. It has not yet been peer reviewed and published, although it is &#8220;under review&#8221; at a well-known journal. So, it&#8217;s technically a &#8220;working paper.&#8221; If you want to get your research geek on, you can check out the link above.<\/p>\n<hr \/>\n<p>Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakc\u0131, O., &amp; Mariman, R. (2024). Generative ai can harm learning.\u00a0<i>Available at SSRN<\/i>,\u00a0<i>4895486<\/i>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I hope that one of my strengths as a blogger is: I know what I don&#8217;t know &#8212; and I don&#8217;t write about those topics. While I DO know a lot about cognitive science &#8212; working memory, self-determination theory, retrieval practice &#8212; I DON&#8217;T know a lot about technology. And: I&#8217;m only a few miles [&hellip;]<\/p>\n","protected":false},"author":18,"featured_media":3205419,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[240,12],"class_list":["post-3205404","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-lb-blog","tag-generative-ai","tag-retrieval-practice"],"_links":{"self":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/3205404","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/comments?post=3205404"}],"version-history":[{"count":1,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/3205404\/revisions"}],"predecessor-version":[{"id":3205440,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/posts\/3205404\/revisions\/3205440"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/media\/3205419"}],"wp:attachment":[{"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/media?parent=3205404"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/categories?post=3205404"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.learningandthebrain.com\/blog\/wp-json\/wp\/v2\/tags?post=3205404"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}