Stemler’s New Study Finds Educational Interventions Cannot Be “Scaled Up”

It turns out that teaching language arts, math and science to fourth graders is not the same as manufacturing cars on an assembly line. That is, the microeconomics principle of economies of scale—or the cost advantages that businesses get by increasing the scale of production—do not always apply to educational interventions.

Put another way, an intervention that works great in one specific educational setting cannot necessarily be “scaled up” to work in many other settings.

This is the finding of a major new study funded by the National Science Foundation, on which Associate Professor of Psychology Steven Stemler collaborated with colleagues at a number of other universities including Yale, Cornell and the University of Sydney. The study, carried out in 223 classrooms across the country in the early- to mid-2000s, was published in the American Psychology Association’s Journal of Educational Psychology in August. The paper is titled “Testing the Theory of Successful Intelligence in Teaching Grade 4 Language Arts, Mathematics, and Science.” The researchers set out to test educational interventions grounded in the theory of Successful Intelligence (SI). This theory acknowledges that different people have different strengths in areas such as creativity, problem solving, or practical skills. In SI-based interventions, teachers appeal to all these different areas in order to connect with different students where they are strongest.

SI interventions have been shown to be effective in a number of smaller studies in the past, but they had never before been scaled up in this way. The new study involved 7,702 fourth-grade students in 223 classrooms in 113 elementary schools in 35 towns located in nine states (Alabama, California, Connecticut, Massachusetts, Minnesota, Kansas, North Carolina, South Carolina, and West Virginia). Stemler became involved in the study while he was an associate research scientist and assistant director of the PACE Center at Yale in the mid-2000s. After the active part of the study in classrooms was completed, many of the researchers left Yale for other institutions, and Stemler was in charge of coordinating the ongoing data analysis with his now-disparate colleagues. When the results came in, the researchers were surprised to find that, in most instances, students receiving SI-based interventions did not perform any better than students in the other control groups. There were even fewer instances where students in the various control groups (which represented different styles of teaching intervention) outperformed students in the SI group. “The main conclusion is: context matters,” said Stemler.

“An educational intervention that works in one place isn’t necessarily going to work in another. Even if you can control for the variables inside the classroom, such as the textbook, curriculum, and class size, there are a host of other variables outside the classroom that have an impact.These factors include the level of poverty in a school, its leadership, experience and buy-in of teachers, background of students, and parental involvement, among others. This shows that trying to apply economic theories and factory-based models to education may be fundamentally misguided.”

One important implication of the study’s findings is in the approach taken by teacher training programs.

“Some programs train teachers in a standardized one-size-fits-all-way,” Stemler explained.

Others, such as Boston Teachers’ Residency, train teachers to work in a very specific context, such as public schools in Boston. Stemler’s research suggests that the more context-focused programs will likely be more effective.