R3 2.7 April 15, 2024 New Study Casts More Doubt on Strict Deadlines; How Quizzes Spark New Learning
Retrieval practice helps you remember information you haven’t even seen yet. But the precise reasons for this remain a mystery.
One of the other newsletters I pay a lot of attention to is Becky Supiano’s, which she writes as part of her reporting for the Chronicle of Higher Education. The newsletter comes out every Thursday; it typically contains a mix of teaching-related research and resources, current news, and questions for readers about the topics she’s currently working on. One recent issue also discussed a new study on a topic close to my heart: myths, misconceptions, and alternative approaches to enforcing deadlines in college courses.
If you’ve followed my own writing for the Chronicle or my blog over the last few years, you know that I am solidly on the flexibility side of the deadline debate. I have become especially disenchanted with the practice of deducting a percentage of points as a function of lateness. Even more so than no-exceptions, no-late-work policies do, grade penalty schemes waste students’ time and attention, while also muddying the meaning of the final grades they end up getting.
The study discussed in Supiano’s newsletter deserves the attention it’s getting. I’ve now gone over it myself, and was impressed with the alignment between the study design and purpose. I completely agree that the results call into question many of the assumptions that faculty make about what will happen if they do accept late work with few or no penalties attached.
In brief, the authors set up an “extension without penalty” system in a large, introductory-level STEM course, which gave all students the chance to turn in work after the suggested deadline without the need for permission or documentation. A survey revealed overall positive impressions of the policy, and there were no decrements in grades associated with having access to or actually using the policy.
One caveat is that the policy studied in this particular project represented a fairly moderate form of flexibility. While there weren’t limits on how many times students could use the extension system, there were still “extension deadlines” that, from my reading at least, put a hard stop on how much extra time a student could have for any given assignment. That said, it’s impressive that even a modest rollback of the typical late-work policy was so roundly endorsed by students, who cited the impacts on stress levels in particular. Intriguingly, even those who did not use the extensions (22% of all students, in total) appreciated the stress reduction associated with the policy. And first-generation students, in particular, also reported that the policy helped encourage them to seek help, e.g., from the teaching assistants in the course or by attending office hours.
If deadlines are a hot button issue for you or the faculty you work with, I’d definitely suggest checking out the original study, and I hope it sparks further research on this important and often-overlooked aspect of course design.
Now, on to another new study, also on a topic that I write and speak about a lot: retrieval practice. Fans of evidence-based teaching know that quizzing for learning is about the biggest discovery ever to come out of applied memory research and instructional design. The principle that pulling information out of memory promotes later recall is well established, but researchers are continuing to investigate the reasons for this effect, as well as trying to identify specific ways of using it that could help students get even more mileage out of retrieval practice as a study technique.
One variation that I’ve found particularly intriguing is test-potentiated learning, which I’d broadly define as improvements to retention for studied material brought about by having just taken a quiz on similar material. This pattern, which is sometimes called the forward testing effect, powerfully demonstrates the importance and scope of what retrieval can do for the memory side of learning. Today’s article digs into the mechanisms that underlie the forward testing effect, and although the authors don’t pin all of these down in the end, the results extend the concept to a new subject matter area and help set the stage for further investigations.
Citation:
Ahn, D., & Chan, J. C. K. (2024). Does testing potentiate new learning because it enables learners to use better strategies? Journal of Experimental Psychology: Learning, Memory, and Cognition, 50(3), 435–457.
DOI:
https://doi.org/10.1037/xlm0001233
Paywall or Open:
Paywall, but will be open-access starting in November 2024
Summary:
Through five separate experiments, this study explores whether retrieval practice potentiates new learning by causing learners to change their learning strategies. Participants described their strategies at different points during a learning task in which they studied Chinese characters and their English translations. Participants were assigned to conditions in which they either re-studied the materials or were tested after initial presentation of the lists of characters to be learned. There were also patterns involving radicals (components of the written characters) that carried over from character to character; transfer of learning was assessed by examining whether participants could use these recurring patterns to correctly guess the meaning of characters they had not studied previously. Across all five experiments, there was strong evidence of forward testing/test potentiated learning, in which performance on new materials improved as a function of having engaged in testing rather than restudying in the recent past. However, the improvements did not correlate with changes to reported study strategies, nor was there evidence of enhanced transfer associated with testing.
Research Questions:
Does testing improve future performance by stimulating changes in learning strategies over time? For example, does testing encourage students to organize associated items or use visualization for future material they are about to study?
Sample:
Undergraduate students (Experiments 1, 3, 4, and 5) or adults recruited via an online platform (Experiment 2). Sample size for Experiments 1-5 respectively: 73, 197, 367, 73, and 183. Participants were English speakers who were unfamiliar with written Chinese.
Method/Design:
All experiments were conducted online, and asked English-speaking participants to learn to identify and translate Chinese written characters. The materials were presented in a sequence of lists, interspersed with either a test or an opportunity to restudy/review the materials. There was then either a cumulative test or a “transfer test” involving a new character that participants had not seen before, but which incorporated a common feature (i.e., a radical) that appeared on previously studied items. Interspersed with the tests were questions asking participants to report what strategies they were using.
Experiments 1-3 randomly assigned participants to either the test condition or restudy condition. In Experiments 1 and 3, they were asked about their strategies after each list; in 2, 4, and 5, they were asked only at the end of all of the lists and before the cumulative or transfer test.
Experiment 2 included conditions with and without feedback. Experiment 3 additionally contrasted results when participants were asked three separate times about the strategies they were using, versus when they were asked only once.
Experiments 4 and 5 used a within-subjects design, to better isolate strategies that a given participant might use under different circumstances.
Key Findings:
The forward testing effect, where being exposed to a test improved performance on subsequent lists, was found across all of the different conditions. However, results did not support the idea that this happened because of strategy change, as there were no differences in reported strategies or patterns of change over the course of the study between the test and restudy conditions. Strategy also did not significantly mediate the forward testing effect. Lastly, the transfer test results did not differ across the test and restudy conditions.
Choice Quote from the Article:
In the present experiments, we tested the strategy-change account for the forward testing effect. Across five experiments, we consistently found no effect of testing on participants’ encoding strategy reports when they attempted to learn written Chinese characters— despite observing a robust forward testing effect in every comparison. Our study also showed that interim testing did not promote the transfer of learning to novel Chinese characters relative to restudying.
Why it Matters:
First and foremost, this project goes to show that retrieval practice is a complicated phenomenon. As more instructors become more aware of retrieval as a learning mechanism, and as it continues to enter the public consciousness through books like the modern classic Make it Stick, it becomes easy for all of us to take for granted this remarkable discovery, or to see testing as something that would obviously help accelerate memory. As these findings show, taking short intermediate quizzes in the run-up to a higher-stakes assessment is still a powerful way to perform better, even in an incredibly challenging arena such as (for English speakers) learning to identify Chinese characters. However, there are probably a number of different processes that contribute to this improvement, and the overall mechanism for how it happens is far from straightforward.
While the findings didn’t reveal a simple story about why the forward testing effect happens, the researchers did an admirable job at ruling out one possibility: that testing causes learners to strategically adjust their approaches for organizing and reviewing information as they encounter it. This possibility can be broadly characterized as a “metacognitive” explanation, given that it has to do with noticing and adjusting study strategies on the fly, and presumably, becoming more adept at this process as part of developing study skills.
I don’t take these new results to mean that metacognition is irrelevant as part of retrieval practice, or to studying in general. I think there is good evidence that, over a longer period of experience with retrieval practice, students begin to notice its effectiveness and adopt it as part of their own study habits. I think that testing does also help students realize where they need to concentrate additional effort, and reduce overconfidence that sets them up for failure down the line. As we go along in the effort to figure out all the mechanisms underlying retrieval practice, I predict that these two factors – overconfidence and metacognition – are going to be important, even if they don’t explain why the forward testing effect happens over the short term.
Most Relevant For:
Instructional designers; foreign language instructors; researchers interested in retrieval practice and memory processes
Limitations, Caveats, and Nagging Questions:
In a puzzling departure from earlier research findings, this study didn’t find that retrieval practice promoted transfer. In other words, quizzing along the way didn’t help participants notice and use the patterns involving radicals that could help them infer the meaning of characters they hadn’t seen before. However, I’m not sure this finding justifies walking back the claim that retrieval practice benefits transfer in general. The way that researchers measured transfer in this particular study, while reasonably valid, might not resemble the kind of transfer that instructors tend to grapple with in other disciplines, which tends to involve application of overall principles or bringing previously learned skills to bear on new problems. As the authors acknowledge, the to-be-transferred material was studied only a few times in advance of the transfer question. Given that the number of repetitions or study opportunities does tend to have a big impact on the likelihood of transfer, this aspect of the design might have obscured any advantage associated with the retrieval practice condition.
If you liked this article, you might also appreciate:
Arnold, K. M. & McDermott, K. B. (2013). Test- potentiated learning: Distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 940–945.
Carpenter, Shana, K., Pan, S. C., & Butler, A. C. (2023). The science of effective learning with spacing and retrieval practice. Nature Reviews Psychology.
Chan, J., Meissner, C. A., & Davis, S. D. (2018). Retrieval potentiates new learning: A theoretical and meta-analytic review. Psychological Bulletin, (July). https://doi.org/10.1037/bul0000166
Karpicke, J. D., Butler, A. C., & Roediger, H. L. (2009). Metacognitive strategies in student learning: Do students practise retrieval when they study on their own? Memory, 17(4), 471–479. https://doi.org/10.1080/09658210802647009
Kenney, K. L., & Bailey, H. (2021). Low-stakes quizzes improve learning and reduce overconfidence in college students. Journal of the Scholarship of Teaching and Learning, 21(2), 79–92. https://doi.org/10.14434/josotl.v21i2.28650
Miller, M.D. (2022). Remembering and Forgetting in the Age of Technology: Teaching, Learning, and the Science of Memory in a Wired World. West Virginia University Press.
Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin. https://doi.org/10.1037/bul0000151
Pennebaker, J. W., Gosling, S. D., Ferrell, J. D., Apfel, N., & Brzustiski, P. (2013). Daily online testing in large classes: Boosting college performance while reducing achievement gaps. PLoS ONE, 8(11), e79774. https://doi.org/10.1371/journal.pone.0079774
Thomas, R. C., Weywadt, C. R., Anderson, J. L., Martinez-Papponi, B., & McDaniel, M. A. (2018). Testing encourages transfer between factual and application questions in an online learning environment. Journal of Applied Research in Memory and Cognition. https://doi.org/10.1016/j.jarmac.2018.03.007
File under: retrieval practice; memory; foreign language instruction; metacognition; study strategies
Do you think the deadline extension could work for daily in-class assignments, too? For instance, a policy that made in-class prompts available after the end of the class?