Ace it! Math Efficacy Study

The Ace it! math efficacy study found that students made modest but significant gains in math. These gains suggest that Ace It! students are catching up to their national peers.

Math Study Details
The following summarizes the results of an evaluation focused on the effectiveness of Sylvan Learning’s Ace it! math program in Pennsylvania, Virginia, and Texas, based on students’ performance on the Group Mathematics Assessment and Diagnostic Evaluation (GMADE), a nationally-normed test published by Pearson AGS Publishing. This study involves longitudinal student data and in some instances combines multiple cohorts of data within each state. Because the test scaling is equivalent across the pre/post administrations of the GMADE, we can examine student improvement over the course of the year (or tutoring program). We can also compare the growth seen in Ace it! students to national norms for the grade level. The following scores from the GMADE are presented in this report:

  • Growth Scale Value (GSV), a developmental standard score that is used to track students’ growth between tests and across grade levels);
  • Percentile ranking (compared to the norm group;
  • Stanine (1–3 = below average; 4–6 = average; 7–9 = above average).

Ace it! Efficacy: Pennsylvania
The charts display the pre-test and post-test scores on the GMADE for the Pennsylvania Ace it! students (2006-07 and 2007-08 cohorts; N=31 students, grades 3–6). The students attended between 32 and 49 hours of Ace it! tutoring, with a mean of 42 hours. Ace it! students made substantial gains in math performance (8.4 GSV gain, 20.4 percentile point gain, and 1.5 stanine gain, on average) from pre-test to post-test. Mean gains in GSV, percentile, and stanine were statistically significant based on a series of paired sample t-tests.

While gains in mean scores provide substantial evidence of the effectiveness of the Ace it! math program, parents are interested in the odds of the program helping their individual child. For that reason, we examined the percentage of Ace it! students that made substantial improvement and found that overall, 71% of students improved their percentile rankings in math by at least 10 percentile points.

For purposes of this report, “substantial improvement” is defined as improving one’s percentile ranking by at least 10 percentile points.

 

Ace it! Efficacy: Texas
The charts display the pre-test and post-test scores on the GMADE for the Texas Ace it! students (combining Spring 2006, Summer 2006, Spring 2007, and Summer 2007 cohorts; N=144 students, grades 2– 8). The students attended between 20 and 48 hours of Ace it! tutoring, with a mean of 30 hours. We see that Ace it! students made significant gains in math performance (2.9 GSV gain, 7.5 percentile point gain, and .5 stanine gain, on average) from pre-test to post-test. Mean gains in GSV, percentile, and stanine were statistically significant based on a series of paired sample t-tests. Examining the percentage of Ace it! students that made substantial improvement and from pre-test to post- test, we found that overall (combining all four cohorts), 43% improved their percentile rankings in math by at least 10 percentile points.

 

Ace it! Efficacy: Virginia
The charts display the pre-test and post-test scores on the GMADE for the Virginia Ace it! students (Spring 2007 cohort; N=15, grades 4–5). The students attended between 13 and 30 hours of Ace it! tutoring, with a mean of 27 hours. Despite the relatively low number of mean tutoring hours attended, we see that Ace it! students made substantial gains in math performance (6.4 GSV gain, 17.7 percentile point gain, and 0.8 stanine gain, on average). These mean gains were statistically significant based on a series of paired sample t-tests. Examining the percentage of Ace it! students that made substantial improvement and from pre-test to post-test (in this case, a 5-month period from January 2007 and May 2007), we found that 77% improved their percentile rankings in math by at least 10 percentile points.

 

Ace it! Efficacy: Combined Sample
This section presents results for the combined sample including all Pennsylvania, Texas, and Virginia Ace it! math program GMADE data (N=190 students). As discussed earlier, because the test scaling is equivalent across the pre / post administrations, we can examine student improvement over the course of the year (or tutoring program). We can also compare the amount of growth seen in Ace it! students to national norms for the grade level. As seen in the charts below, students participating in Ace it! math programs in Pennsylvania, Texas, and Virginia made modest improvement across the three performance indicators from pre-test to post-test, with mean gains of 4.0 GSV, 10.5 percentile points, and 0.7 stanine. These gains were statistically significant based on a series of paired sample t-tests. Overall, the results of this evaluation indicate that Ace it! students are catching up to their national peers (i.e., improving percentile rank by 10 points on average) and provide preliminary evidence of the efficacy of the Ace it! math program.

These results are considered preliminary, in that we do not have GMADE scores for a control group of non-Ace it! students attending the same schools as our Ace it! sample. Future efficacy studies will involve comparisons to such a control group, which will allow us to draw stronger conclusions about the effect of the Ace it! program on student gains compared to the effect of regular school-day instruction alone. Future efficacy studies will also include an analysis of state achievement test data (for states that use vertically scaled assessments that make it possible to calculate student growth/improvement across grade levels). Such an analysis was not possible for the current evaluation of Ace it! within PA and VA because these states use horizontally scaled assessments (comparable only within grade levels).