Heavy Lifting - thoughts and web finds by an economist
     I also contribute to Division of Labour Load HL's Front Page
Wednesday, August 16, 2006

Is the ACT score increase economically significant?

The 2006 ACT results have been released and the news is supposedly good. While more students took the ACT than in the past, the average composite score increased 0.2 points over 2005. One wonders if this is a statistically significant result and whether it is an economically significant result. Moroever, how can this result be used by those who want to justify even more spending in K-12, but mainly in secondary education? It is difficult to know without the standard deviation of the test scores, so I moved to a second-best approach.

I gathered the data on average ACT score by state from 1994 through 2006. The data included the average score and the percentage of students who took the test. Two states during the sample period began requiring all students to take the ACT (Colorado and Illinois). I created a dummy variable for those states that required the ACT and regressed average score against REQUIRED, the percentage of students that took the test, and yearly dummy variables. I used a fixed effects estimator to control for state-specific unmeasured heterogeneity (of which there is a lot!).

Here's what we get:

Fixed-effects (within) regression Number of obs = 663
Group variable (i): stateid Number of groups = 52

R-sq: within = 0.2940 Obs per group: min = 1
between = 0.0142 avg = 12.8
overall = 0.0552 max = 13

F(14,597) = 17.75
corr(u_i, Xb) = -0.1600 Prob > F = 0.0000

------------------------------------------------------------------------------
avescore | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
required | -.8270902 .1311565 -6.31 0.000 -1.084674 -.5695059
pertested | -.0120328 .0025851 -4.65 0.000 -.0171098 -.0069559
_Iyear_1995 | -.0026525 .0490132 -0.05 0.957 -.0989118 .0936069
_Iyear_1996 | .0640552 .0490345 1.31 0.192 -.0322459 .1603564
_Iyear_1997 | .0754857 .0490769 1.54 0.125 -.0208987 .1718702
_Iyear_1998 | .2054748 .0492517 4.17 0.000 .1087471 .3022025
_Iyear_1999 | .1890075 .0492779 3.84 0.000 .0922284 .2857866
_Iyear_2000 | .192904 .0498993 3.87 0.000 .0949045 .2909036
_Iyear_2001 | .144519 .0500077 2.89 0.004 .0463066 .2427313
_Iyear_2002 | .141248 .0504037 2.80 0.005 .0422578 .2402382
_Iyear_2003 | .2291622 .0504896 4.54 0.000 .1300034 .328321
_Iyear_2004 | .3470181 .0514054 6.75 0.000 .2460606 .4479755
_Iyear_2005 | .3814437 .0514333 7.42 0.000 .2804314 .482456
_Iyear_2006 | .4970106 .0515253 9.65 0.000 .3958176 .5982035
_cons | 21.38062 .1060703 201.57 0.000 21.1723 21.58893
-------------+----------------------------------------------------------------
sigma_u | 1.0434525
sigma_e | .2474407
rho | .94676015 (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(51, 597) = 185.59 Prob > F = 0.0000

How to read these results? Those states that required their students to take the ACT averaged 0.82 points lower in average score. This is not surprising as the population taking the test is considerably larger, forcing many students who would not have taken the test (presumably because they would not have scored as well, or were otherwise less inclined - perhaps because they weren't planning on attending college) to take the test.

Thus the true impact of, say, Colorado requiring the ACT is more accurately calculated by looking at both the impact of the REQUIRED variable (-0.827) and the impact of the higher percentage of students taking the test.

In 2003 the ACT became required in Colorado. In 2002, 99% of students took the ACT. However, in 2001, only 62% took the test (indeed for the sample years before 2001, the average was 63% taking the test in Colorado. Therefore, the requirement to take the ACT also added an approximately 38 additional percentage points to the number taking the test. Thus, the true impact of requiring the ACT was a -0.827 - 38x(-0.012) = -1.283 decline in the average score. Of course, we still don't know anything about the standard deviation, but it is anticipated that this would also increase. Thus, it is difficult to say anything about the coefficient of variation (mean divided by the standard deviation).

On to the time trends, which are represented by the _Iyear_XXXX variables. These coefficients indicate how the average score in the United States, after controlling for the percentage taking the test, whether a state required the ACT, and unmeasured state differences, differed from the average score in 1994.

For instance, in 1997, the average score was 0.075 points higher than in 1994 but the difference was not "statistically significant." In other words, we can't say with reasonable confidence, say at the 95% level, that the parameter 0.075 is actually different from zero. On the other hand, in 2000 the average score was 0.19 points higher than in 1994 and this value is statistically different from zero (notice the P>|t| - the p-value - is less than 0.05).

Indeed, the average score in 2006 was approximately 0.50 points higher than it was in 1994. As the average scores continue to approach the perfect score of 36, it is unlikely that the yearly changes would continue to increase, that is they will have to decline eventually if the trend towards perfect scores is to continue. Nevertheless, from the difference from 1994 in 2005 (0.38) and the difference from 1994 in 2006 (0.497) annual change from 2005 to 2006 can be determined as 0.497-0.381 = 0.115 points which is statistically different from zero (P-value of 0.019).

The ACT report claims an unconditional increase of 0.2 points, whereas my conditional results, admittedly not conditional on much, suggest a 0.115 point change. However, if testing whether 0.115 is statistically different from 0.2, yields a p-value of 0.087. This suggests that at the 95% confidence level we cannot claim that 0.115 is different from 0.2. However, at the 91% confidence level we can. I'll let the reader decide which way to go.

Notwithstanding some interesting time trends, the omitted variables, such as per-pupil spending, number of teachers, teacher-student ratio, etc., are numerous and can cause considerable problems. To some extent, the fixed effects estimator can account for many of these issues, but including more explanatory variables is generally better.

I gathered per-pupil current dollar spending for the years available (1994-2004) and converted the spending into 2004 dollars using the December CPI. Including real per-pupil spending in the original model yields the following results:
. 
Fixed-effects (within) regression Number of obs = 561
Group variable (i): stateid Number of groups = 51

R-sq: within = 0.2578 Obs per group: min = 11
between = 0.0267 avg = 11.0
overall = 0.0333 max = 11

F(13,497) = 13.28
corr(u_i, Xb) = -0.4166 Prob > F = 0.0000

------------------------------------------------------------------------------
avescore | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
required | -.5884529 .169992 -3.46 0.001 -.9224444 -.2544614
pertested | -.0203382 .0029101 -6.99 0.000 -.0260558 -.0146205
rspending | .0000225 6.95e-06 3.24 0.001 8.84e-06 .0000362
_Iyear_1995 | -.0001523 .0465078 -0.00 0.997 -.0915284 .0912237
_Iyear_1996 | .0556733 .0465439 1.20 0.232 -.0357738 .1471204
_Iyear_1997 | .0756951 .0466578 1.62 0.105 -.0159757 .167366
_Iyear_1998 | .2043557 .0470868 4.34 0.000 .111842 .2968694
_Iyear_1999 | .194675 .0469617 4.15 0.000 .1024071 .2869429
_Iyear_2000 | .2222634 .0476835 4.66 0.000 .1285773 .3159495
_Iyear_2001 | .1671894 .04788 3.49 0.001 .0731172 .2612616
_Iyear_2002 | .1665644 .0484513 3.44 0.001 .0713699 .261759
_Iyear_2003 | .2486486 .0487175 5.10 0.000 .152931 .3443663
_Iyear_2004 | .3714066 .0501063 7.41 0.000 .2729604 .4698528
_cons | 21.52622 .126796 169.77 0.000 21.27709 21.77534
-------------+----------------------------------------------------------------
sigma_u | 1.0459665
sigma_e | .23476357
rho | .9520398 (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(50, 497) = 179.53 Prob > F = 0.0000

What do we see here? The parameter estimates on REQUIRED and PERTESTED change a little, suggesting that including real spending ed is useful. Indeed, more spending does correlate with higher average scores. The parameter estimate suggests that each additional (real) dollar spent per-pupil yields an increase in the average ACT score in the state of 0.000025 points.

The measure of spending used here does not qualify the type of spending - that is, is it on technology, teachers, administrators, junkets to Hawaii - but let's be real, this is a blog entry not a writeup for the American Economic Review.

There are likely all sorts of endogeneity problems with the specification, that is funding might be related to test scores, and vice versa, but we will address this in turn (see below).

As it appears there is a positive relationship between spending and test scores, many will be quick to suggest that we spend more and continue to increase test scores. Yet, there is likley some concavity in the relationship between spending and test scores, mainly because there is a natural limit to the test score (at 36). By including the square (quadratic) of per-capita spending we can test for curvature and estimate the amount of spending that would maximize average ACT scores.

Including the quadratic we obtain a parameter estimate on per-pupil spending of 0.0000956 and the parameter estimate on the quadratic of -1.33E-09. Using these two parameters, it is possible to estimate that amount of spending, conditional on the other variables in the model, that would maximize the average ACT score.

Sans the fancy calculus, this value is 0.0000956/(2x1.33E-09) = 35,953 (2004 dollars) per student. This is clearly not feasible in the current economic and political environment and even if it were it is likely that such spending would carry too high an opportunity cost. This suggests that maximizing the average test score is not likely to be a successful goal (although that won't stop people from trying).

Just to be complete, it is possible to estimate what the average ACT score would be, given that 60% of the students take the test voluntarily and spending was at $36,000:

21.10298 - 60*0.0219 + 36000*0.0000956 - 36000*36000*1.33E-09 = 21.82

In other words, the maximized average (given our results here) would be 21.82 points and the 95% confidence interval would be [21.30, 22.35]. This suggests that additional spending would have little economic impact on average ACT scores (although this is only one dimension of focus, there might be a lot of other issues that additional spending might influence in a more dramatic fashion).


There are concerns about the endogeneity of spending as it is likely related to test scores. Using fancy econometrics, I estimated a dynamic panel data model based on the Arellano-Bond specification. These results yield:
Arellano-Bond dynamic panel-data estimation, one-step difference GMM results
------------------------------------------------------------------------------
Group variable: stateid Number of obs = 510
Time variable : year Number of groups = 51
Number of instruments = 6 Obs per group: min = 10
F(4, 505) = 16.27 avg = 10.00
Prob > F = 0.000 max = 10
------------------------------------------------------------------------------
| Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
pertested | -.0225992 .0042967 -5.26 0.000 -.0310407 -.0141576
required | -.0545356 .1952975 -0.28 0.780 -.4382313 .3291601
rspending | .0004299 .000054 7.97 0.000 .0003239 .0005359
rspending2 | -7.11e-09 9.69e-10 -7.34 0.000 -9.02e-09 -5.21e-09
------------------------------------------------------------------------------
Sargan test of overid. restrictions: chi2(2) = 22.06 Prob > chi2 = 0.000

Arellano-Bond test for AR(1) in first differences: z = -3.22 Pr > z = 0.001
Arellano-Bond test for AR(2) in first differences: z = -6.12 Pr > z = 0.000
I am trying the patience of the reader, I understand, but bear with me for just a little longer, if you have thus far. These results suggest that there might be a considerable bias in the impact of real spending on average ACT scores from 0.00006 to 0.0004 points return from the marginal dollar spent (560%!).

However, the dynamic panel data estimates suggest that average ACT scores might be maximized with approximately $30,000 (2004 dollars) per pupil and that the maximized . Although the bias is large, the ultimate conclusion that it is unlikely that we can spend enough to maximize average scores is not overturned.

Back to the original question. Is the increase in the ACT scores economically significant. My answer is that is not tremendously significant on its own. However, the long-term trend that the scores are increasing might be good news and suggests that the spending on education are not completely wasted.

[STATA data file]

Comments: Post a Comment



Purchase




Le Chai - galerie du vin



Support



Popularity

Posts that contain Craig Depken per day for the last 90 days.



Contacts

Heavy Lifting's Main Page
Email Me
Atom Feed

WWW
Heavy Lifting




Great Links

Money I Found Today

Heavy Lifting - Firehose style (56k warning)



Recent Posts

- Is the ACT score increase economically significant?




Archives


home


Visitors
Site Meter Blogroll Me!



Credits

Modified maystar design
powered by blogger