Ace it! Reading Efficacy Study

The Ace it! reading efficacy study found that students made modest but significant gains in reading. These gains suggest that Ace It! students are catching up to their national peers.

Reading Study Details
The following summarizes the results of an evaluation focused on the effectiveness of Sylvan Learning’s Ace it! reading program in Pennsylvania and Virginia, based on students’ performance on the Group Reading Assessment and Diagnostic Evaluation (GRADE), a nationally normed test published by Pearson AGS Publishing. This study involves longitudinal student data and in some instances combines multiple cohorts of data within each state. Because the test scaling is equivalent across the pre/post administrations of the GRADE, we can examine student improvement over the course of the year (or tutoring program). We can also compare the growth seen in Ace it! students to their peers (national norms for the grade level). The following scores from the GRADE are presented in this report:

  • Growth Scale Value (GSV), a developmental standard score that is used to track students’ growth between tests and across grade levels;
  • Percentile ranking (compared to the norm group); and
  • Stanine (1–3 = below average; 4– 6 = average; 7–9 = above average).

Ace it! Efficacy: Pennsylvania
The charts display the pre-test and post-test scores on the GRADE for the Pennsylvania Ace it!
students (2006-07, 2007-08, and 2008-09 cohorts; N = 109, grades 3–6). The students attended between 15 and 49 hours of Ace it! tutoring, with a mean of 41 hours. Depending on the metric examined, we see that Ace it! students made modest to substantial gains in reading performance (18.1 GSV gain, 6.6 percentile point gain, and 0.4 stanine gain). Mean gains in GSV, percentile, and stanine were statistically significant based on a series of paired sample t-tests. While gains in mean scores provide evidence of the effectiveness of the Ace it! reading program, parents are interested in the odds of the program helping their individual child. For that reason, we examined the percentage of Ace it! students that made substantial improvement and found that overall 46% improved his/her percentile ranking in reading by at least 10 percentile points.

For purposes of this report, “substantial improvement” is defined as improving one’s ranking by at least 10 percentile points.

Ace it! Efficacy: Virginia
The charts display the pre-test and post-test scores on the GRADE for the Virginia Ace it! students (Spring 2007 cohort; N = 29, grades 4–5). The students attended between 19 and 30 hours of Ace it! tutoring, with a mean of 27 hours. Despite the relatively low number of mean tutoring hours attended, we see that Ace it! students made substantial gains in reading performance (8.8 GSV gain, 10.1 percentile point gain, and 0.7 stanine gain, on average). These mean gains were statistically significant based on a series of paired sample t-tests. Examining the percentage of Ace it! students who made substantial improvement from pre-test to post-test (in this case, a 5-month period from January 2007 and May 2007), we found that 52% improved their rankings in reading by at least 10 percentile points.

Ace it! Efficacy: Combined Sample
This section presents results for the combined sample including all Pennsylvania and Virginia Ace it! reading program GRADE data (N = 138 students). As seen in the charts below, students participating in Ace it! reading programs in Pennsylvania and Virginia made modest improvement across the three performance indicators from pre-test to post-test, with mean gains of 16.2 GSV, 7.3 percentile points, and 0.5 stanine. These gains were statistically significant based on a series of paired sample t-tests. Overall, the results of this evaluation suggest that Ace it! students are catching up to their national peers (i.e., improving percentile rank by 10 points on average) and provide preliminary evidence of the efficacy of the Ace it! reading program.

These results are considered preliminary in that we do not have GRADE scores for a control group of non-Ace it! students attending the same schools as our Ace it! sample. Future efficacy studies will involve comparisons to such a control group, which will allow us to draw stronger conclusions about the effect of the Ace it! program on student gains compared to the effect of regular school-day instruction alone. Future efficacy studies will also include an analysis of state achievement test data (for states that use vertically scaled assessments that make it possible to calculate student growth/improvement across grade levels). Such an analysis was not possible for the current evaluation of Ace it! within PA and VA because these states use horizontally scaled assessments (comparable only within grade levels).