American Institutes for Research

Over two years, AIR partnered with teachers and students across the country to conduct five evaluation studies of the CueThinkEF+, Fraction Ball, and MathFluency+ products. These studies were designed to examine the impact that math interventions which integrate executive function skill development have on students’ math skills, EF skills, and perceptions and beliefs about mathematics. Through quantitative and qualitative data collection, AIR investigated implementation variation across contexts, relationships between product use and student outcomes, and teacher and student perspectives of and experiences with each product. Taken together, the study findings provide a comprehensive picture of how these learning approaches support students and teachers in the mathematics classroom.

AIR used innovative study designs that allowed them to measure evidence of impact while also providing actionable insights related to product implementation. These novel evaluation designs differed from more traditional approaches, as they uncovered timely data around new product features that were being tested, while also measuring product effectiveness. For CueThinkEF+, the team used a Sequential Multiple Assignment Randomized Trial (SMART) design to rapidly assess the relative effects of four different adaptive versions of the CueThinkEF+ platform. The following year, AIR conducted a randomized controlled trial with an embedded factorial design to simultaneously test six different combinations of features on the CueThinkEF+ platform. Together, these innovative methods allowed AIR to identify which product features work, and for whom, more quickly than conventional evaluation cycles.

EF+Math set a vision and collaborated with AIR, as their approach to evaluation positioned educators, students, and product development teams as true partners in research. AIR established District Advisory Boards, which included district representatives who were involved in all stages of the evaluation, from study design to data interpretation. AIR ensured that their evaluation process centered educator expertise, and aligned research questions with district priorities, incorporated educator input when making study decisions, and provided opportunities to interpret, contextualize, and make meaning of study findings in collaboration with educators. The end result is evaluation studies whose research questions, data, and findings are more actionable and useful to districts, since district needs and priorities have been considered throughout every stage of the process.

In order to conduct these evaluation studies, new asset-based measures were co-created by AIR and the R&D teams to assess key outcomes such as students’ problem solving skills and students’ attitudes and beliefs about mathematics. In collaboration with CueThinkEF+, AIR created a problem solving assessment that evaluates both students’ problem solving strategies, as well as computational accuracy. AIR also developed a Student Math Beliefs and Attitudes Survey that measured students’ self-efficacy, math identity, math anxiety, confidence in math, and belonging in the math classroom. Together, these tools allowed AIR to capture students’ growth in the problem solving skills and positive mathematical perceptions that are essential to long-term success.

Through its active management approach, the EF+Math program set a goal for its evaluation studies to be conducted using a set of aligned, common measures, in order to enable further exploration of the program’s hypothesis. As a result, AIR was able to create an integrated dataset of executive function skill, mathematics, and student math perception outcomes, using data from 10,000 students in grades 3-8 who participated in the evaluations of CueThinkEF+, Fraction Ball, and MathFluency+. This valuable resource, combining quantitative data at multiple time points with standardized assessment data and demographic data, enables the further exploration of relationships between EF skills, math learning, and student perceptions across diverse products, contexts, and studies. The creation and analysis of this dataset demonstrates how a portfolio-approach to evaluation can generate insights and learnings that would be impossible to achieve through individual studies alone.