Special Report: Assessment Interrupted
Implications of COVID-19 for Accountability in 2021 and How to Progress
Since the 90s, state testing has served as the primary measure of effectiveness in K-12 education. Results of student achievement on standardized tests are incorporated into accountability models, a rubric for evaluating school and district performance, in the majority of states across the nation. Many states’ accountability models are then written into law, meaning amendments and reformations to the model must pass legislation.
This prominence of state testing is one of the reasons it was disruptive when states were able to suspend or waive all required testing due to COVID-19. Though necessary, this interruption to testing introduces some practical complications. Beyond their use in accountability models, test scores are critical in helping educators report achievement and growth, support low and underperforming students, and evaluate progress toward goals. This article discusses these challenges and offers suggestions for moving forward.
Measuring achievement and growth is as important today as it was pre-COVID-19.
Test results provide insight on student knowledge and performance, critical information for determining where to begin instruction. Articles about learning loss during the coronavirus pandemic have already circulated. The Brooking Institution used data from prior catastrophic disruptions and warned that students may show less than 50% of learning gains from the prior year in math due to extended school closure. The Annenberg Institute at Brown University found similar results in their models, and project that students will likewise return to school with approximately 65% of reading gains observed in a typical year.
Though you may not have results of state tests to reference, most students will have taken two (fall and winter) benchmarks during the 2020 school year. Benchmark data will be helpful, if not critical, for indicating where to begin instruction, and aggregate results can paint a picture of achievement for your school or district. Benchmarks are designed to estimate students’ performance on higher-stakes tests, and projections typically have acceptable rates of accuracy. For highly vetted assessments, correlations between predicted and observed scores generally range from .50 - .70, with some variation by subject and grade level. Unless your students were testing on a brand new benchmark for the very first time, you can place a lot of confidence in your benchmark’s results.
When they do occur, discrepancies in benchmark-predicted scores and observed scores usually lean conservative and underestimate rather than overestimate performance. Students’ performance on benchmark assessments tends to improve over the course of the year, meaning that fall scores will likely underestimate performance to a greater degree than will winter scores. These conservative tendencies will be directly applicable in a year when official results are unavailable and large losses in learning are expected.
As we transition into the next school year, the NWEA’s number one recommendation for supporting growth and achievement is to establish authentic relationships with students and address their social and emotional needs first and foremost. Next, they recommend addressing content that was skipped and work to bridge the gap for 2021 grade-level standards. The assessment organization along with researchers at The Annenberg Institute, Brookings, and Renaissance also recommend that educators be prepared to handle differentiated instruction, as “students are likely to enter school with more variability in their academic skills than under normal circumstances.”
What could this mean for assessment and accountability outcomes in 2021? Results may not be as dire as predicted, but prepare to see lower end-of-year scores than normal. Imagine students are taking a brand new assessment this year or envision every single grade going through a key transition year as if all students collectively felt the shock of leaving middle school and entering high school. Renaissance recommends “gathering professional learning communities beyond a grade level” to mitigate academic skill variability and bridge the gap of key content that was skipped in 2020.
Exacerbated equity and opportunity gaps increase the urgency to support low-performing students.
Early identification of high-risk students is particularly critical this year since modeling suggests exacerbated gaps and learning losses for this group of students. Specifically, there is evidence that students from disadvantaged backgrounds have unequal access to technology, intensifying the negative impact of online compared to classroom learning (Maslen, 2020). According to The 74, this disparity means that “the lowest-achieving kids may fall two more years behind” in a single classroom and that teachers can expect to see more students on the extreme ends of the performance spectrum this school year.
Betebenner and Van Iwaarden argue that “growth analyses can be of significant assistance in identifying the differential educational impact of the COVID-19 pandemic.” They advocate the use of student growth percentile analyses, as they’re “ideally suited to quickly investigate the impact of disruptions to student learning.” They also recommend using fall 2020 interim assessments to determine whether some groups were more affected by early school closure than others.
Early identification is helpful only if the information is used to intervene with appropriate support and remediation. Dr. Tequilla Brownie, executive vice president of the nonprofit consulting firm The New Teacher Project, warns against lowering expectations, however. She stresses the importance of administrators working with teachers to ensure students are given the chance to perform grade-appropriate work. “Too often”, she says, “not enough students are [getting access to grade-appropriate assignments], and overwhelmingly that number comprises students of color, students from low-income families, students who have mild to moderate learning disabilities, and English language learners” (Zalaznick, 2020, p. 4).
Dr. Brownie explains that students’ performance exceeds expectations when they’re given the chance to engage in rigorous academic work. This claim is supported by consistent empirical correlational evidence from the Organisation for Economic Cooperation and Development’s (OECD, 2016) periodic testing program, Programme for International Student Assessment (PISA). High academic standards for all students and creating supportive learning environments to meet this demand are among the organization’s recommendations for tackling low performance.
Prepare for a significant multi-year impact on accountability.
COVID-19’s disruption to accountability and assessment will expand beyond the 2020 and 2021 school years, says Chris Domaleski of the Center for Assessment. He explains that two primary reasons for this multi-year disruption are (1) the ubiquity of academic growth in accountability models and its requirement of prior measurements or multi-year averaging and (2) attempts to substitute or repair data “will substantially change the meaning and interpretation of results” (Domaleski, 2020, April 7). Though not impossible to overcome, these challenges aren’t trivial, and flexibility beyond 2021 will be necessary.
Domaleski (2020) asserts that it’s unreasonable to expect states to return to the pre-coronavirus accountability model. Ultimately, this disruption to testing presents a rare opportunity “to invest in accountability innovations” that otherwise wouldn’t have occurred, says Domaleski. He advocates for changes that unfold in phases that are “friendly to improvement and innovation.”States will need time to analyze data and work with advisors and policymakers as they re-examine foundational assumptions about their model. “To be clear,” he says, “even the most earnest and aggressive attempts to restore legacy accountability systems will not produce results that can be meaningfully compared to those prior to 2020.”
This process of reassessing accountability, performing analyses, and outlining options may take two years (Gong et al., 2020). If you’re curious about this process on the state leadership level, Gong and colleagues (2020) created a semester-by-semester checklist for state accountability and assessment leaders. Potential areas of improvement for accountability models discussed by Domaleski include more explicit connections to school improvement needs, which “requires more fine-grained information throughout the system to be able to target interventions where they are most needed” (Marion & Evans, 2020). Another potential adjustment is coherence between both the state’s goals and meeting federal requirements outlined for college and career readiness (Landl, 2018). Coherence and balance between federal, state, and local accountability was also suggested. Domaleski also recommends considering implementing school-level performance expectations that adhere to credible standards with relevant consequences and criteria (Domaleski et al., 2018).
Whether or not states seize the opportunity to address improvements, it’s clear that school accountability will change and it will take some time. Patience is a virtue, and in this season it may as well be a friend. You and your colleagues will all need to remind each other to be patient while the state grapples with some rather complex and nuanced challenges that have wide-ranging consequences. It’s okay for this to take some time. Talk to your assessment vendors about any concerns that you may have; they’ve been in the testing business a long time and will have some insight on how to make the most of your data. Work closely with administrators and faculty to ensure that there’s a plan of action and everyone is working toward the same goals.
Though patience will be key to dealing specifically with accountability, now is not the time to be idle. As stated previously, building authentic relationships with students is critical. A test score cannot tell you whether a student excels at minecraft or reacts emotionally to music. Help faculty get to know students and their families so they can focus on connecting content in meaningful ways. If there’s a common theme in the challenges and opportunities discussed in this post, it’s growth- it is mentioned in each section of this article. Of course, I was referring to academic growth in each instance. I’ll discuss growth this time in reference to relationships. Focus on growing relationships and ensuring students’ needs are met.
This work can begin immediately and will have a positive impact on students’ success. Start connecting in meaningful ways and getting to know the students who fill your classrooms, hallways, gymnasium, cafeteria, and stadium- those places where we soon hope to be together. We know from the work of Joyce Epstein “when schools work together with families to support learning children tend to succeed not just in school, but throughout life” (Olivos et al., 2011, p. 11). Having solid relationships with parents will be critical for supporting students as we navigate the upcoming school year under the constraints of coronavirus.
References
Betebenner, D., & Van Iwaarden, A. (2020). Issues and considerations that the COVID-19 pandemic presents for measuring student growth. Discussing the implications of calculating student growth scores for accountability and using growth as a tool to analyze the impact of school closure on student learning. Center for Assessment, April 15.
COVID-19 Policy Considerations: Accountability (2020). How to deal with canceled assessments. The Intersection, The Hunt Institute Blog, April 1.
Cowan Pitre, C. (2014). Improving African American student outcomes: Understanding educational achievement and strategies to close opportunity gaps. Western Journal of Black Studies, 38, 209 - 217.
Cowen, J. (2020). Closures, testing, and accountability: What coronavirus means for schools. Forbes, March 13.
Domaleski, C. (2020). The outlook for ESSA school accountability after COVID-19. A return to the status quo is unlikely and that’s just fine. Center for Assessment, April 7.
Domaleski, C. (2020). School disruption due to COVID-19. A high-level overview of likely implications and options for assessment and accountability. Center for Assessment, March 13.
Domaleski, C., D’Brot, J., Keng, L., Keglovits, R., Neal, A-M. (2018). Establishing performance standards for school accountability systems. Council of Chief State School Officers, July.
Dorn, S. (2020). The pandemic and cultural scripts of school-family relationships. Teaching and Learning During a Pandemic, Albert Shanker Institute, June 11.
Education Commission of the States (2018). 50 state comparison. Accountability and reporting: Current system. http://ecs.force.com/mbdata/mbQuest6S?rep=SA171
Gong, B., Marion, S., & Domaleski, C. (2020). Dealing with fallout from COVID-19 school disruptions: What to do next in assessment and accountability? States need to deal with the present, plan for the future. Center for Assessment, April 16.
Hargreaves, A. (2020). What’s next for schools after coronavirus? Here are 5 big issues and opportunities. Shanker Blog, Albert Shanker Institute, May 6.
Hawkins, B. (2020). New research predicts steep COVID learning losses will widen already dramatic achievement gaps within classrooms. The 74, June 9.
Honda, M. (2010). Interview- Bridging gaps in education and the community. American Teacher, 94, p. 5.
Honored to be a chosen in the category of Best Collaborative Community Solution
