How to Measure What Matters
Photo by Dennis Eusebio on Unsplash.
Before we dive into the topic, I want you to pause and close your eyes. Think about the data dashboards you have at work. These are the fancy data graphics that show high stakes metrics like enrollment totals, end-of-year test scores, suspension rates, and attendance rates.
Now shift your thinking towards the information you need to make daily decisions. Are there dashboards for those?
If you’ve been following along with the exercise, you probably realized what I did: There’s no “Dashboard for Whether My Students Are Learning to Read” website or “Update on Whether My Students Are Learning Fractions” report.
“If a mechanic recommends you replace all of your spark plugs because your speedometer said 78 miles per hour or because your check engine light turned bright red, you might want a second opinion.”
This doesn’t mean dashboards are inherently bad. They’re not. Trust me on this—I make them for a living. It just means that they’re dashboards and nothing more. Like an actual dashboard in a car, they alert you to something you should pay attention to. But most of the time they don’t tell you what’s actually happening and they certainly don’t tell you why something is happening.
Think about it this way: If a mechanic recommends you replace all of your spark plugs because your speedometer said 78 miles per hour or because your check engine light turned bright red, you might want a second opinion.
Big Unclear Questions Are Made Up of Small Clear Questions
So what do we do when the dashboards we have don’t answer the practical questions we have? We break our questions down to smaller questions. Then we see if we have data that can help us answer those.
Let’s look at a non-education example. Let’s say you want to have a fun evening with friends. And let’s say you want to develop a method that shows you if your evening with friends was actually fun.
Unfortunately, there’s no existing “fun evening meter” dashboard you can use. But you can reasonably deconstruct “a fun evening” into its separate parts. There’s all kinds of ways to do this, so you’ll need to apply your own knowledge. But a fun evening could include:
A great dinner
Good conversation
Live music
Even these are ambiguous, so let’s deconstruct them further. For example, let’s take “a great dinner.” A great dinner might include:
Food that your friends like
A quiet atmosphere
Reasonable prices
Now this is more concrete. While “A fun evening” feels ambiguous, “reasonable prices” feels a little clearer. You can set a threshold for a reasonable dinner price, then check to see if dinner costs more than that. And you can make a mental note of whether your friends could hear each other in the restaurant, which helps you judge a quiet atmosphere.
I mean sure these folks look like they’re having a good time, but is there a dashboard for that? Photo by Rendy Novantino on Unsplash.
All this probably seems like overkill for the purposes of a planning an evening out, but the example illustrates a real data problem in education: the data you receive is not always the data you need.
So invest a little effort defining your data question, and make sure it’s broken down into small clear parts. You’ll find it easier to make the connection between measurement and real life decisions.
Examples
Next time you need to answer an ambiguous question like “Are students learning content from the math lesson?” or “Do students feel safe at school?” Start by breaking that question down into smaller parts. And keep breaking it down into smaller parts until you get to something you can measure.
For example, here’s how you might start with a broad question and end up at something more measurable. Notice how I push for smaller parts. Sometimes I find it helpful to ask the question “How would I know” over and over until I get to something measurable:
Are students learning content from the math lesson > Do they understand the concept > Can they talk about it clearly > Do I hear them talking about it when I observe the activities?
And finally, here are some cool examples of smaller but measurable activities, just to get your thought process going:
Ask students to write everything they know about a topic on a mini white board
Observe activities and see how students justify their answers, reproduce their examples, and reflect on their learning
Establish learning targets for students, then use available data to check progress towards them
Use exit slips or reflection activities at the end of the lesson
Notes
Most examples I use in this article were inspired by the following online resources:
Barefoot TEFL Teacher. “How Do You Check Your Learners are Learning?” Barefoot TEFL Teacher, https://www.barefootteflteacher.com/blog/check-your-learners-are-learning. Accessed 26 September 2021.
Webb, Helen. “How do you know your students are learning?” SecEd, 1 March 2017, https://www.sec-ed.co.uk/best-practice/how-do-you-know-your-students-are-learning/. Accessed 26 September 2021.
Burton, Amy. “How do I know my students are learning?” The Learning Forward Journal, April 2020, https://learningforward.org/journal/beyond-the-basics/how-do-i-know-my-students-are-learning/. Accessed 26 September 2021.
