So Does Data in Schools Even Matter?
Photo by Daniel Mingook Kim
I spend a lot of time with education data. Like, a lot. If I’m not talking about it, writing about it, or trying to convince people to use it, I’m probably in a room with my pug on a Sunday just thinking about it. Note the date of this post.
But as a fan of inquiry, evidence, and certainty, I’m regularly visited by a nagging thought: how much does data use actually matter when it comes to making school really great for students?
Searching for the Answer
In 2017, three researchers, Jeffrey Wayman, Shana Shaw, and Vincent Cho, had the same question. But unlike me, they chose to actually answer it instead of just thinking themselves in circles on a Sunday. And so together, they set out to answer the question: Can teachers influence student learning by looking at data reports?
“I’m regularly visited by a nagging thought: how much does data use actually matter when it comes to making school really great for our students?”
This is not an easy thing to study. The research conditions would have to be just right. But the brave researchers assembled all the ingredients to try: a school district in the Southwestern United States, a software system that teachers used for periodic assessments, and an agreement to observe how it all went down for two years.
Wayman, Shaw, and Cho counted how many times teachers in the school district used the software system and its related data. Over a two year period they pored through the software use logs of hundreds of teachers and the achievement tests of thousands of students.
Here’s what they found: in all but one instructional area, there was no relationship between how often the teachers used the data reports and how well students did on the academic tests.
What It Means for Educators
Crisis averted. So glad I read the rest of the article.
I read this and had a moment of panic. How we use data in schools, a topic that I care about deeply, might not matter. I considered curling up in the fetal position with my pug as I pondered the meaning of my professional existence.
But instead, I read the rest of Wayman, Shaw, and Cho’s article. And I’m glad I did.
It turns out using data in schools is not just about looking at data reports. It’s a deeply human endeavor that requires more than that. Wayman, Shaw, and Cho point to human factors for school communities to consider as they learn to use data. What may matter more than how often educators look at data is what they do with it.
“It turns out using data in schools is not just about looking at data reports. It’s a deeply human endeavor that requires more than that.”
They point to studies about all the good and bad associations educators have with data. They point to studies about how data can be used to intentionally shape the craft of educators. And they point to studies about the training and coaching leaders need to put in place as an essential part of a school district’s data culture.
Here’s what we can take away from this study: we can’t just look at data and expect for things to change for our students. We’ve got to push past looking and into the hard work of learning. And then as we learn what works in our school district’s context, we need to change our practice based on what we learned. Or as Wayman, Shaw and Cho put it: “Technology is never the agent of change—people are. When data systems are implemented and applied in this way, we are optimistic that they can pay off on the promise relative to the investment.”
Notes
You can read the full research article here: Wayman, Jeffrey, Shaw, S., and Cho, V. “Longitudinal Effects of Teacher Use of a Computer Data System on Student Achievement,” AERA Open, 2017, retrieved 1 Aug. 2021. https://files.eric.ed.gov/fulltext/EJ1194175.pdf
Thank you to Dr. Alex Bowers who showed this paper to me almost a year ago while helping me with my work as a Strategic Data Project fellow. You should check his stuff out too, which is really awesome.
