How to Measure Success Without Academic Achievement

This piece was originally published by EdSurge on November 22, 2017.

The question of how to measure success in life is one typically left to philosophy classes or late nights at the bar. It is a complex, perhaps unanswerable question. In the words of the cast of Rent, how do you measure a life?

So it’s really a wonder that we consider the definition of success for personalized learning programs to be so simple. Can the success of any educational initiative be measured by academic achievement alone?

This June, a subset of the education world was upended when RAND released a report detailing the components of personalized-learning implementations and the effects this model has on students. The report’s data showed that students had made only modest academic achievement gains on math and ELA assessments, and these findings unleashed a massive wave of skepticism, with many people claiming that the model doesn’t work or didn’t live up to the hype.

As the blended learning coordinator for the Notre Dame ACE Academies, a multi-state network of Catholic schools serving some of the nation’s most underserved students, I understand, and absolutely believe in, the importance of measuring the success of educational initiatives with academic achievement. As I wrote in a recent piece for the Christensen Institute, my colleagues and I will not implement any program at the expense of student learning, and we carefully track our students’ growth and achievement to ensure success. But we don’t stop there, and neither should you.

For us, the value proposition of personalized learning extends beyond academic achievement, and therefore the measure of its success does the same. In fact, we believe that the true magic lies in the potential to increase certain critical noncognitive factors of success for participating students.

The Whole Person Approach

Students in our schools have two goals—college and heaven—and the education we offer has to be designed to help students meet both of these goals. (If you don’t believe in God or the afterlife, replace “heaven” with “being a good person.”) The bottom line is that our criteria for success for any initiative is that it must help students become better learners and better human beings, and we are attempting to measure the success of our programs by the growth students experience in both of these dimensions.

Because I would argue that every initiative should be measured in terms of its impact on students’ academic and personal growth, I want to outline three steps you can take to design a program that develops students as whole persons.

1. Define ‘being a good person’

Successful programs name the skills or strengths they want to help develop. You don’t want every single one of your students to have the exact same set of character traits, but you do want every student to have a certain set of characteristics that you consider to be especially important. Identify those characteristics so that you can design for them (see step 2).

When we decided to pilot a new learning initiative in classrooms across multiple schools in our network, we knew that we wanted it to align with our five core values: seek, persist, excel, serve and love. Though all five values are deliberately cultivated in various ways, we chose to focus the design of our personalized learning program in particular on the first three values: seek, persist and excel.

We want our students to seek the answers to their own questions or problems, to understand the role they play in their own success and actively pursue it, and to make a conscious effort to be the best versions of themselves. We want them to persist when they encounter tough problems or personal challenges and to know that effort will trump ability every time. And of course we want them to excel, both in academic and nonacademic settings.

2. Design your program to cultivate these characteristics

If you assume that implementing “personalized learning” (whatever that means to you) will make your students independent, self-directed, intrinsically motivated learners, you will be sorely disappointed. Fostering these characteristics and others in students requires deliberately designing a program with these characteristics as the intended outcomes. For example, if you want your students to develop a growth mindset, what will you and your students do in response to failure?

We created a number of different systems and routines in our classrooms in order to help our students seek, persist, and excel. One of these systems is goal setting, a practice in which students engage every week. In our schools, students as young as second grade seek to accomplish goals by identifying what they want to learn that week and what specific actions and behaviors will be required to achieve it. And when students fail, we help them learn from that experience by identifying what prevented them from succeeding and what they could have done differently to achieve the desired result. 

With the appropriate systems in place, failure becomes an opportunity to learn more about the barriers to success the students encounter every day and how to persist when they encounter those barriers in order to ultimately accomplish their goals.

3. Determine how to measure growth—then do it!

Though it is typically simpler to measure academic achievement, it is not impossible to measure the growth of noncognitive skills and we have to be better about trying. Organizations like the Character Lab and the University of Chicago Consortium on School Research have developed measures of character strengths such as grit, growth mindset and internal locus of control that can be used for scientific research. For more informal uses, student focus groups and student or teacher surveys will likely get the job done. Regardless of which measures you choose, you must select or design tools to measure the growth of the skills you want students to develop. How else will you truly know whether or not your program is successful?

We cannot claim to be experts on measuring students’ noncognitive skills, but we do have carefully designed tools and systems in place that constitute our attempt to measure the success of our program beyond the overly-simplistic metric of student test scores. Using measures from the Character Lab, as well as surveys and interview protocols designed by our own research and evaluation team, we collect regular data on the growth of specific skills we want to see in our students.

Though the data we have so far is limited, it has already helped us make adjustments to some practices and double down on others. For example, last year we found that students were not persisting in their online lessons when they failed—instead, lesson failure often resulted in pouting, reduced effort, or perhaps an attempt to abandon work for the day altogether. To address this issue, we instituted a system of reflection that students undertake every time they fail an online lesson that includes a written self-reflection on why the failure occurred as well as a student-teacher conference in which both the student and teacher identify action steps they can take to ensure the student does not fail the lesson again.

This process is not easy, but that is likely part of the reason why it has significantly reduced the number of lessons students fail and increased the number of lessons they successfully complete. And though this process does have an impact on academic achievement, it was borne of our evaluation of our students as whole persons and will ultimately have its most significant impact on students’ noncognitive skills as well.

The vast majority of us believe that educational programs should have an impact on more than students’ academic achievement, but the problem is that we rarely measure the success of our programs on metrics other than test scores. The key now is to start a broader conversation about what constitutes a successful learning program—personalized or otherwise—and identify metrics we can use to measure that success. Only then will be able to fully capture the magic of new, transformative ways of learning.