We are all investors

It was time for a new elementary math curriculum in Traverse City, Michigan, and the school district decided to take a pretty unconventional approach to making its selection.

This is no small matter, as purchasing curricula (including print or digital textbooks, workbooks, and other components) can cost millions of dollars, and districts typically only make this investment every five to ten years.

In Traverse City, the curriculum adoption committee had narrowed it down to three new curricula, each of which was backed by research. But here’s where things get interesting: the district then decided to run a year-long pilot study of all three curricula and include a control group of students who would continue to use the existing materials.  Principals, teachers, and district leaders ran the pilot together.

At the end of the year, they found that only two of the curricula produced statistically significant improvements. They could then compare the financial costs of the two products, and they had practical wisdom from teachers who had implemented each product in the classroom to inform the decision about which product to select, what components to purchase, and how to roll it out to the rest of the district.

The associate superintendent overseeing math instruction called it “the best experience of my career.” One school board member shared, “For the first time in my board tenure, I feel that decisions have been rooted in objective information.”

This is one example of Academic Return On Investment (A-ROI), a collection of practices that many school districts are adopting to make more strategic decisions about how to invest their funds and how to evaluate the impact of their programs.

The ABCs of A-ROI

I’ve been learning about A-ROI from sources including the Government Finance Officers Association, the District Management Group, and Education Resource Strategies.

The question at the heart of A-ROI is: What does the most good, for whom, and at what cost?

Districts are using A-ROI to adopt new programs and initiatives, like in the math curriculum example shared above. Often, they run limited pilots before they scale implementation across a whole district.

Districts also use A-ROI to evaluate the return on investments they’ve already made, ensuring that existing initiatives are worth the time, money, and effort being expended.

Because staffing comprises the largest part of any school district’s budget, it’s important to capture the amount of staff time a particular program or initiative requires, as part of its overall cost; this is challenging, but not impossible, to do. There are formulas, tools, and templates available from districts that have already begun this journey.

That said, because A-ROI is intense, districts can’t analyze everything. Often, they choose to focus on the programs that consume the most resources, or where they’ve identified that a number of programs overlap and there might be redundancy.

The “Ugly Christmas Tree” in Boulder Valley, Colorado

A few years ago, Boulder Valley School District  was struggling with the same problem that a lot of school districts face: in a well-intentioned effort to support as many students as possible, it had layered one initiative on top of another, creating what  one former district leader calls “the ugly Christmas tree” effect: “too many decorations that, while individually well-intended, don’t work well together and weigh down the very thing they were intended to support.”

An initiative inventory confirmed some suspicions: school staff were trying to implement 251 initiatives from 28 teams across nine departments in the central office. Over the next six months, the district worked to glean as much information as it could about

• the students served by each program
• its known outcomes
• its fully loaded costs, including allocation of staff time
• and its connection to other efforts.

In parallel, through a survey of school principals district leaders gauged their perceived value of each program, and for which students. They also asked principals about the implementation status of each program, and whether additional support was needed to implement it effectively.

This didn’t instantly fix the problem—but it gave Boulder Valley a good place to start. The district is using this inventory to create a roadmap for when and how it will conduct more thorough analyses of specific initiatives as a regular part of its ongoing operation.

Five Tips I’ve Learned From Districts Who’ve Done It

1. Be clear at the outset about what “success” looks like. When a new initiative is proposed, specify the outcomes that will be measured, by whom, and when. Make sure everyone knows what data would be considered proof of success later on.

2. Combine evidence-based decision making with cost-benefit analysis. Evidence-based decision making says “Wow! This program delivers great results!” Cost-benefit analysis says, “Yeah, but it costs sixty gazillion dollars per student. What if we could get 70% of that same benefit with a program that costs a little less, and allows us to work on this other instructional need, too?”

3. Don’t be afraid of pilot tests. I’ve said in a school board meeting, “We can’t run pilot tests” and here is where I eat my words. We can and probably should. It’s the best way to reduce the risk of a district spending too much of its money and students’  and teachers’ time on an intervention that doesn’t work.

4. Beware the sunk cost fallacy. Only the likely future benefits and costs of a program—not the sunk costs—should be considered when making a decision on whether to invest in a program going forward. I think of this as the “bad boyfriend” cognitive bias. Yeah, you’ve been with him for four years. You’ve invested a lot of effort. But girl, it’s still time to go.

5. Don’t make it just about cutting costs. A-ROI might yield budget savings, but it’s ultimately about making the best instructional decisions for the district’s students. As such, it’s a process a district should run completely separately from its budget development cycle. Some districts also adopt formal policies stating that no employee will lose their job as a result of the findings of an A-ROI analysis. That analysis can lead to any one of the following results:

• Wow. Great value. Let’s expand this program.
• Delivering really well for some student populations: let’s use it in more targeted ways.
• Results aren’t clear: let’s continue to monitor for X period of time.
• We’ve uncovered this flaw: let’s fix that flaw and reevaluate in X period of time.
• Let’s abandon this program.

 
In my work as a school board member, I can appreciate how intensive A-ROI would be to implement. But I am triply certain that A-ROI or a discipline very much like it is absolutely essential. We have to exercise this discipline if we are to be good stewards of taxpayer dollars, if we want to avoid overburdening educators with low- or no-value initiatives, and—most important—if we are really committed to providing the best education to our communities’ youngest citizens.