# Calculator 1

The practical significance of achievement gaps is difficult to grasp, when group mean differences are expressed in original test scores (as measured by scale score differences) or in standard deviation units. Given that the pace of academic growth gets slower and slower at the higher ages/grades, what are the implications of such diminishing growth rates for any intervention efforts to reduce achievement gaps? We can translate the gaps in school year units using information on annual academic growth rates.

In an experimental or quasi-experimental research design using both a pretest and a posttest, academic growth between the two time points is estimated using the control group gain as the basis of a time-indexed effect size calculation. Figure 1 illustrates the concept and measurement of time-indexed effect size based on hypothetical linear patterns of growth for an experimental group (E) and a control group (C). Assuming that both groups have the same average pretest scores, Ye and Yc represent the average posttest scores of the achievement variable Y for the experimental group and control group respectively.

Unlike a conventional effect size metric such as Cohen's d that measures the group difference on the vertical axis (achievement variable in standard deviation units), we shift the focus of measurement to the horizontal axis (time variable in school year/month units). The time-indexed effect size (d΄) is the extra time (in school years/months) needed for the control group to reach Ye, the level of outcome that the experimental group has reached at the end of treatment (see Figure 1 below).

d΄ = T2 – T1

Where...
T2 = time needed for the control group to reach Ye;
T1 = time spent for the experimental group to reach Ye or for the control group to reach Yc Figure 1

This notion of time-indexed effect size analysis can be extended to nonexperimental/observational research where preexisting group differences in achievement are measured. For example, the national average achievement gap between White and Black students in grade 4 math is about .9 in standard deviation units (i.e., 90% of a standard deviation). This difference is deemed large by Cohen's effect size benchmarks (.2 for small, .5 for medium and .8 for large). What does this statistical interpretation mean to educators in practical school terms? How long would it take for Black students to catch up to the current performance level of White peers? If we evaluate the achievement gap from the viewpoint of “time needed to catch up”, we need to know the average Black student’s annual growth rate first. Since the average math achievement growth rate is estimated to be 0.7 standard deviation units for fourth grade Black students, the White-Black gap of d = 0.9 is worth about 1 year and 3 months of schooling time; d΄ = 0.9 divided by 0.7 = 1.3. The calculator below allows you to convert Cohen's d (standardized group mean differences) into d΄ (years/months of schooling, i.e., time-indexed effect size) for any type of achievement gap at any grade in reading and math.

## Time-indexed Effect Size for P-12 Reading and Math Research/Evaluation

### Research Team

• PI: Jaekyung Lee (Professor of Education, University at Buffalo)
• Co-PI: Jeremy Finn (Professor of Education, University at Buffalo)

### Abstract

The purposes of this IES-funded research project are (a) to develop academic growth references for P-12 reading and math achievement based on nationally-representative longitudinal datasets; and (b) to develop a time-indexed effect size metric, based on new national norms, that can be used to assess the effectiveness of educational interventions. Conventional effect size metrics such as Cohen’s d are standardized group mean differences based on the distributions of student outcome variables at one particular age or grade level. They do not take into account time dimension, that is, time needed to learn at that age/grade level. This study is based on the premise that time-indexed effect size metrics can estimate how long it would take for an “untreated” control group to reach the treatment group outcome in terms familiar to educators — months of schooling.

#### Full Abstract

The purposes of this IES-funded research project are (a) to develop academic growth references for P-12 reading and math achievement based on nationally-representative longitudinal datasets; and (b) to develop a time-indexed effect size metric, based on new national norms, that can be used to assess the effectiveness of educational interventions. Conventional effect size metrics such as Cohen’s d are standardized group mean differences based on the distributions of student outcome variables at one particular age or grade level. They do not take into account time dimension, that is, time needed to learn at that age/grade level. This study is based on the premise that time-indexed effect size metrics can estimate how long it would take for an “untreated” control group to reach the treatment group outcome in terms familiar to educators—months of schooling. They would differ from conventional grade equivalent (GE) metrics as strength-of-effect measures, which suffer from several limitations. For instance, GEs are drawn from test publishers’ norms derived from cross-sectional data of different cohort groups at a single year to estimate growth curves. Moreover, the assumption under GE that the study sample would grow at the same rate as the national norms could be erroneous. The new measures would adjust the growth trajectory based on national longitudinal data using vertical scales of achievement along with information regarding the demographic profiles of the study sample and settings.

We proposed, a time-indexed effect size metric to estimate the effect size in the unti of school years/months. It serves as a supplement to conventional effect size metrics such as Cohen’s d by taking into account different amounts of time needed for learning at different age or grade levels. Through applications to Project STAR small class effects and NAEP racial achievement gaps, we demonstrate how to interpret and use d΄. Further, we applied time-indexed effect size formula to selected examples of curricular interventions in P-12 that provided information on intent-to-treat (ITT) effect sizes and met evidence standards by What Works Clearinghouse (WWC). It is expected to provide a more developmentally appropriate context for interpreting the size of an effect, a step toward bridging the gap between educational research and practice.

### Research Paper Presentations and Publications

Lee, J. & Finn, J. (2010, June). Developing Time-Indexed Effect Size Metrics for K-12 Reading and Math Educational Evaluation. Poster presented at the Institute of Education Sciences (IES) annual conference, Washington, DC.

Lee, J., Finn, J. & Liu, X. (2011, April). Time-Indexed Effect Size Metric for K-12 Reading and Math Educational Evaluation.  Paper presented at the annual meeting of American Educational Research Association (AERA), New Orleans, Louisiana.

Lee, J., Finn, J. & Liu, X. (2012, March). Time-Indexed Effect Size Metric for P-12 Reading and Math Program Evaluation.  Poster presented at the spring conference of Society for Research on Educational Effectiveness (SREE), Washington, DC.

For quick reference, we constructed a table of conversions (see Table 2 below). Three common benchmark values of Cohen’s d (0.2 for small effect, 0.5 for medium effect and 0.8 for large effect) were converted into years/months of schooling by dividing d values by corresponding standardized yearly gain scores. We followed the same steps to construct the conversion table for demographic subgroups based on their national longitudinal growth norms. Table 2