Table of Contents
Introduction
The purpose of this report is to provide the results of the mixed-effects model building process applied in our Content & Convergent Evidence Study to answer the following research questions.
1a. Are there differences at the passage-level in WCPM between the human scoring criterion versus traditional or ASR scoring of traditional CBM-R and CORE passages (i.e., scoring method)?
1b. Are there differences at the passage-level in WCPM between the traditional CBM-R and CORE passages (i.e., passage length)?
- Are there differences at the passage-level in time duration between the human scoring criterion versus traditional or ASR scoring of traditional CBM-R and CORE passages?
In response to these Research Questions, we used lme4 to apply a mixed-effects model separately for each of Grades 2 through 4 and each outcome, WCPM and time duration (3 grades x 2 outcomes = six modeling processes).
For a full description of the purpose and procedures of the Content & Convergent Evidence Study, go here.
Random Effects
Summary of Random Effects Findings
For both WCPM and time and all Grades 2 through 4, the model with random effects for both students and passages statistically improved the model fit compared to models with a random effect for either students or passages.
WCPM
We began by comparing unconditional models for WCPM and time with and without random-effects for student and passage.
Here are the WCPM results for Grades 2 through 4 for the model with a random effect for students:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
98859.3 98881.2 -49426.6 98853.3 10995
Scaled residuals:
Min 1Q Median 3Q Max
-4.9728 -0.5922 -0.0675 0.5346 9.5737
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1418.1 37.66
Residual 417.3 20.43
Number of obs: 10998, groups: student_id, 261
Fixed effects:
Estimate Std. Error t value
(Intercept) 85.116 2.341 36.36
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
133002.6 133025.3 -66498.3 132996.6 14370
Scaled residuals:
Min 1Q Median 3Q Max
-4.4816 -0.5997 -0.0424 0.5401 9.7405
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1183.7 34.41
Residual 551.9 23.49
Number of obs: 14373, groups: student_id, 323
Fixed effects:
Estimate Std. Error t value
(Intercept) 110.850 1.926 57.57
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
129443.5 129466.1 -64718.7 129437.5 13989
Scaled residuals:
Min 1Q Median 3Q Max
-5.8352 -0.6069 -0.0308 0.5630 9.3287
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1465.0 38.28
Residual 547.7 23.40
Number of obs: 13992, groups: student_id, 315
Fixed effects:
Estimate Std. Error t value
(Intercept) 132.271 2.167 61.05Here are the WCPM results for Grades 2 through 4 for the model with a random effect for passages:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
112593.3 112615.2 -56293.7 112587.3 10995
Scaled residuals:
Min 1Q Median 3Q Max
-2.4192 -0.7312 -0.0687 0.6209 5.1728
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 74.8 8.649
Residual 1607.2 40.090
Number of obs: 10998, groups: passage_id, 107
Fixed effects:
Estimate Std. Error t value
(Intercept) 88.627 0.922 96.13
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
146393.2 146415.9 -73193.6 146387.2 14370
Scaled residuals:
Min 1Q Median 3Q Max
-3.2933 -0.6843 -0.0380 0.6401 5.9000
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 147.5 12.15
Residual 1521.6 39.01
Number of obs: 14373, groups: passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 111.495 1.225 91.05
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
145151.5 145174.1 -72572.7 145145.5 13989
Scaled residuals:
Min 1Q Median 3Q Max
-3.2986 -0.6868 -0.0305 0.6484 4.9736
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 125.9 11.22
Residual 1841.3 42.91
Number of obs: 13992, groups: passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 132.47 1.15 115.2Here are the WCPM results for Grades 2 through 4 for the model with a random effect for both students and passages:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
97118.2 97147.5 -48555.1 97110.2 10994
Scaled residuals:
Min 1Q Median 3Q Max
-5.5168 -0.5670 -0.0334 0.5334 9.7353
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1424.85 37.747
passage_id (Intercept) 80.06 8.948
Residual 344.26 18.554
Number of obs: 10998, groups: student_id, 261; passage_id, 107
Fixed effects:
Estimate Std. Error t value
(Intercept) 84.92 2.50 33.97
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
129517.7 129548.0 -64754.9 129509.7 14369
Scaled residuals:
Min 1Q Median 3Q Max
-5.0945 -0.5666 -0.0190 0.5527 11.0883
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1196.0 34.58
passage_id (Intercept) 147.6 12.15
Residual 418.8 20.46
Number of obs: 14373, groups: student_id, 323; passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 110.847 2.265 48.94
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
126698.1 126728.3 -63345.1 126690.1 13988
Scaled residuals:
Min 1Q Median 3Q Max
-6.0317 -0.5682 -0.0011 0.5730 9.8824
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 1461.2 38.23
passage_id (Intercept) 117.4 10.84
Residual 436.5 20.89
Number of obs: 13992, groups: student_id, 315; passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 132.351 2.405 55.03Model Comparison Tests
We conducted deviances tests, comparing the model with a random effect for students to the model with random effects for both students and passages.
$`Grade 2`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 98859 98881 -49427 98853
.y 4 97118 97147 -48555 97110 1743 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 133003 133025 -66498 132997
.y 4 129518 129548 -64755 129510 3486.8 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 129443 129466 -64719 129437
.y 4 126698 126728 -63345 126690 2747.3 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1And comparing the model with a random effect for passages to the model with random effects for both students and passages.
$`Grade 2`
Data: .
Models:
.x: wcpm ~ 1 + (1 | passage_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 112593 112615 -56294 112587
.y 4 97118 97147 -48555 97110 15477 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: wcpm ~ 1 + (1 | passage_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 146393 146416 -73194 146387
.y 4 129518 129548 -64755 129510 16877 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: wcpm ~ 1 + (1 | passage_id)
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 145151 145174 -72573 145145
.y 4 126698 126728 -63345 126690 18455 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Our conclusion was that the addition of student and passage as random effects statistically improved the model fit.
By evaluating the random effects, we found that 68% to 77% of the variance was between students, while only 4% to 8% of the variance was between passages, and 19% to 24% was residual variance. Thus, most of the variance was between students for these WCPM models. See below for WCPM ICC estimates.
$`Grade 2`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 1425. 0.77
2 var_(Intercept).passage_id passage_id 80.1 0.04
3 var_Observation.Residual Residual 344. 0.19
$`Grade 3`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 1196. 0.68
2 var_(Intercept).passage_id passage_id 148. 0.08
3 var_Observation.Residual Residual 419. 0.24
$`Grade 4`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 1461. 0.73
2 var_(Intercept).passage_id passage_id 117. 0.06
3 var_Observation.Residual Residual 437. 0.22Time Duration
Here are the time results for Grades 2 through 4 for the model with a random effect for students:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
62874.6 62895.3 -31434.3 62868.6 7329
Scaled residuals:
Min 1Q Median 3Q Max
-2.2583 -0.6609 -0.2481 0.4011 7.2226
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 88.23 9.393
Residual 286.41 16.924
Number of obs: 7332, groups: student_id, 261
Fixed effects:
Estimate Std. Error t value
(Intercept) 30.7336 0.6185 49.69
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
79817.2 79838.7 -39905.6 79811.2 9579
Scaled residuals:
Min 1Q Median 3Q Max
-2.0972 -0.6882 -0.2811 0.4024 7.8302
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 60.64 7.787
Residual 225.46 15.015
Number of obs: 9582, groups: student_id, 323
Fixed effects:
Estimate Std. Error t value
(Intercept) 25.4726 0.4614 55.2
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id)
Data: .
AIC BIC logLik deviance df.resid
75069.5 75091.0 -37531.8 75063.5 9325
Scaled residuals:
Min 1Q Median 3Q Max
-2.1465 -0.6976 -0.3711 0.4212 9.1010
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 34.15 5.844
Residual 171.50 13.096
Number of obs: 9328, groups: student_id, 315
Fixed effects:
Estimate Std. Error t value
(Intercept) 21.3739 0.3575 59.79Here are the time results for Grades 2 through 4 for the model with a random effect for passages:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
59003.7 59024.4 -29498.9 58997.7 7329
Scaled residuals:
Min 1Q Median 3Q Max
-3.1058 -0.5891 -0.1893 0.4373 8.8472
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 197.7 14.06
Residual 171.6 13.10
Number of obs: 7332, groups: passage_id, 107
Fixed effects:
Estimate Std. Error t value
(Intercept) 30.427 1.368 22.24
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
71286.8 71308.3 -35640.4 71280.8 9579
Scaled residuals:
Min 1Q Median 3Q Max
-4.3827 -0.5308 -0.1697 0.3302 14.1183
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 180.67 13.44
Residual 94.08 9.70
Number of obs: 9582, groups: passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 25.167 1.309 19.22
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
65285.4 65306.9 -32639.7 65279.4 9325
Scaled residuals:
Min 1Q Median 3Q Max
-3.8478 -0.5187 -0.1711 0.3293 12.7341
Random effects:
Groups Name Variance Std.Dev.
passage_id (Intercept) 140.25 11.843
Residual 60.34 7.768
Number of obs: 9328, groups: passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 21.129 1.153 18.32Here are the time results for Grades 2 through 4 for the model with a random effect for both students and passages:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
54058.8 54086.4 -27025.4 54050.8 7328
Scaled residuals:
Min 1Q Median 3Q Max
-6.1990 -0.4620 -0.0278 0.3851 9.5112
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 104.14 10.205
passage_id (Intercept) 207.99 14.422
Residual 76.09 8.723
Number of obs: 7332, groups: student_id, 261; passage_id, 107
Fixed effects:
Estimate Std. Error t value
(Intercept) 31.151 1.535 20.29
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
64459.5 64488.2 -32225.8 64451.5 9578
Scaled residuals:
Min 1Q Median 3Q Max
-7.5167 -0.4702 -0.0199 0.4023 9.5565
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 73.67 8.583
passage_id (Intercept) 184.60 13.587
Residual 40.04 6.328
Number of obs: 9582, groups: student_id, 323; passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 25.382 1.405 18.06
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: time ~ 1 + (1 | student_id) + (1 | passage_id)
Data: .
AIC BIC logLik deviance df.resid
58507.8 58536.3 -29249.9 58499.8 9324
Scaled residuals:
Min 1Q Median 3Q Max
-6.4615 -0.4766 -0.0312 0.4094 17.7257
Random effects:
Groups Name Variance Std.Dev.
student_id (Intercept) 35.77 5.980
passage_id (Intercept) 141.32 11.888
Residual 25.54 5.053
Number of obs: 9328, groups: student_id, 315; passage_id, 106
Fixed effects:
Estimate Std. Error t value
(Intercept) 21.105 1.204 17.52Model Comparison Tests
We conducted deviances tests, comparing the model with a random effect for students to the model with random effects for both students and passages.
$`Grade 2`
Data: .
Models:
.x: time ~ 1 + (1 | student_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 62875 62895 -31434 62869
.y 4 54059 54086 -27025 54051 8817.8 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: time ~ 1 + (1 | student_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 79817 79839 -39906 79811
.y 4 64460 64488 -32226 64452 15360 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: time ~ 1 + (1 | student_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 75070 75091 -37532 75064
.y 4 58508 58536 -29250 58500 16564 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1And comparing the model with a random effect for passages to the model with random effects for both students and passages.
$`Grade 2`
Data: .
Models:
.x: time ~ 1 + (1 | passage_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 59004 59024 -29499 58998
.y 4 54059 54086 -27025 54051 4946.9 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: time ~ 1 + (1 | passage_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 71287 71308 -35640 71281
.y 4 64460 64488 -32226 64452 6829.3 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: time ~ 1 + (1 | passage_id)
.y: time ~ 1 + (1 | student_id) + (1 | passage_id)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 3 65285 65307 -32640 65279
.y 4 58508 58536 -29250 58500 6779.7 1 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Our conclusion was that the addition of student and passage as random effects statistically improved the model fit.
By evaluating the random effects, we found that 18% to 27% of the variance was between students, while only 54% to 70% of the variance was between passages, and 13% to 20% was residual variance. Thus, the majority of the variance was between passages, which is intuitive because time duration and passage length (word count) are highly and directly correlated.
See below for time ICC estimates.
$`Grade 2`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 104. 0.27
2 var_(Intercept).passage_id passage_id 208. 0.54
3 var_Observation.Residual Residual 76.1 0.2
$`Grade 3`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 73.7 0.25
2 var_(Intercept).passage_id passage_id 185. 0.62
3 var_Observation.Residual Residual 40.0 0.13
$`Grade 4`
# A tibble: 3 x 4
term group estimate icc
<chr> <chr> <dbl> <dbl>
1 var_(Intercept).student_id student_id 35.8 0.18
2 var_(Intercept).passage_id passage_id 141. 0.7
3 var_Observation.Residual Residual 25.5 0.13Fixed Effects
To the model with random effects for student and passages, we then added fixed-effects for passage length (four levels: easyCBM, Short, Medium, and Long) and scoring method (three levels: ASR, Recording, and Traditional), and compared it to a model that included an interaction term for passage length x scoring method.
Summary of Fixed Effects Findings
For both WCPM and time and all Grades 2 through 4, the model with the addition of the interaction effect statistically improved the model fit compared to model without the interaction. Thus, our final model for both outcomes and all grades included random effects for student and passage, and fixed effects for passage length, scoring method, and their interaction.
WCPM
Here are the WCPM results for Grades 2 through 4 for the model with a random effect for students and passages, and fixed effects for passage_length and scoring_method:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
96842.53 96908.28 -48412.27 96824.53 10989
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 37.753
passage_id (Intercept) 8.831
Residual 18.309
Number of obs: 10998, groups: student_id, 261; passage_id, 107
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
92.367 -12.898 -10.224
passage_lengthshort modewcpm_criterion modewcpm_traditional
-11.383 4.060 7.226
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
128999.85 129068.01 -64490.93 128981.85 14364
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 34.59
passage_id (Intercept) 11.76
Residual 20.09
Number of obs: 14373, groups: student_id, 323; passage_id, 106
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
110.857 -7.909 -7.357
passage_lengthshort modewcpm_criterion modewcpm_traditional
-1.782 3.872 9.413
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
126025.12 126093.04 -63003.56 126007.12 13983
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 38.23
passage_id (Intercept) 10.40
Residual 20.38
Number of obs: 13992, groups: student_id, 315; passage_id, 106
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
118.226 3.719 8.612
passage_lengthshort modewcpm_criterion modewcpm_traditional
11.433 3.958 10.959 And here are the WCPM results for Grades 2 through 4 for the model with a random effect for students and passages, and fixed effects for passage_length, scoring_method, and their interaction:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
96751.95 96861.53 -48360.97 96721.95 10983
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 37.755
passage_id (Intercept) 8.835
Residual 18.221
Number of obs: 10998, groups: student_id, 261; passage_id, 107
Fixed Effects:
(Intercept)
92.647
passage_lengthlong
-13.272
passage_lengthmedium
-10.437
passage_lengthshort
-11.682
modewcpm_criterion
15.717
modewcpm_traditional
-5.271
passage_lengthlong:modewcpm_criterion
-10.998
passage_lengthmedium:modewcpm_criterion
-10.977
passage_lengthshort:modewcpm_criterion
-12.702
passage_lengthlong:modewcpm_traditional
12.119
passage_lengthmedium:modewcpm_traditional
11.612
passage_lengthshort:modewcpm_traditional
13.598
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
128959.43 129073.03 -64464.72 128929.43 14358
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 34.59
passage_id (Intercept) 11.76
Residual 20.05
Number of obs: 14373, groups: student_id, 323; passage_id, 106
Fixed Effects:
(Intercept)
112.670
passage_lengthlong
-9.746
passage_lengthmedium
-8.907
passage_lengthshort
-3.785
modewcpm_criterion
10.634
modewcpm_traditional
-2.783
passage_lengthlong:modewcpm_criterion
-5.522
passage_lengthmedium:modewcpm_criterion
-6.941
passage_lengthshort:modewcpm_criterion
-7.291
passage_lengthlong:modewcpm_traditional
11.030
passage_lengthmedium:modewcpm_traditional
11.588
passage_lengthshort:modewcpm_traditional
13.293
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
125976.36 126089.55 -62973.18 125946.36 13977
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 38.23
passage_id (Intercept) 10.40
Residual 20.33
Number of obs: 13992, groups: student_id, 315; passage_id, 106
Fixed Effects:
(Intercept)
117.626
passage_lengthlong
5.271
passage_lengthmedium
9.673
passage_lengthshort
11.496
modewcpm_criterion
13.904
modewcpm_traditional
2.812
passage_lengthlong:modewcpm_criterion
-9.767
passage_lengthmedium:modewcpm_criterion
-9.913
passage_lengthshort:modewcpm_criterion
-10.273
passage_lengthlong:modewcpm_traditional
5.109
passage_lengthmedium:modewcpm_traditional
6.728
passage_lengthshort:modewcpm_traditional
10.082 Model Comparison Test
We conducted a deviance test, comparing the model with main effects only to the model with main effects and their interaction.
$`Grade 2`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 9 96843 96908 -48412 96825
.y 15 96752 96862 -48361 96722 102.58 6 < 2.2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 9 129000 129068 -64491 128982
.y 15 128959 129073 -64465 128929 52.421 6 1.534e-09 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: wcpm ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 9 126025 126093 -63004 126007
.y 15 125976 126090 -62973 125946 60.762 6 3.151e-11 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Our conclusion was that the addition of the passage_length x scoring_method statistically improved the model fit. Thus, our final WCPM included random effects for students and passages, and fixed effects for passage_length, scoring_method, and a passage_length x scoring_method interaction.
Time
Here are the time results for Grades 2 through 4 for the model with a random effect for students and passages, and fixed effects for passage_length and scoring_method:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
53746.78 53801.98 -26865.39 53730.78 7324
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 10.141
passage_id (Intercept) 3.956
Residual 8.690
Number of obs: 7332, groups: student_id, 261; passage_id, 107
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
55.163 3.546 -19.186
passage_lengthshort modesecs_traditional
-33.755 -1.512
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
64126.45 64183.80 -32055.23 64110.45 9574
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 8.530
passage_id (Intercept) 3.419
Residual 6.309
Number of obs: 9582, groups: student_id, 323; passage_id, 106
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
55.435 -5.847 -25.952
passage_lengthshort modesecs_traditional
-39.970 -1.044
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode
Data: .
AIC BIC logLik deviance df.resid
58103.73 58160.85 -29043.86 58087.73 9320
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 5.974
passage_id (Intercept) 2.160
Residual 5.036
Number of obs: 9328, groups: student_id, 315; passage_id, 106
Fixed Effects:
(Intercept) passage_lengthlong passage_lengthmedium
55.7996 -13.8480 -31.4367
passage_lengthshort modesecs_traditional
-43.2396 -0.8251 And here are the time results for Grades 2 through 4 for the model with a random effect for students and passages, and fixed effects for passage_length, scoring_method, and their interaction:
$`Grade 2`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
53730.10 53806.00 -26854.05 53708.10 7321
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 10.141
passage_id (Intercept) 3.957
Residual 8.676
Number of obs: 7332, groups: student_id, 261; passage_id, 107
Fixed Effects:
(Intercept)
52.493
passage_lengthlong
6.758
passage_lengthmedium
-16.491
passage_lengthshort
-31.123
modesecs_traditional
3.826
passage_lengthlong:modesecs_traditional
-6.423
passage_lengthmedium:modesecs_traditional
-5.389
passage_lengthshort:modesecs_traditional
-5.263
$`Grade 3`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
64093.15 64171.99 -32035.57 64071.15 9571
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 8.531
passage_id (Intercept) 3.420
Residual 6.296
Number of obs: 9582, groups: student_id, 323; passage_id, 106
Fixed Effects:
(Intercept)
52.626
passage_lengthlong
-2.721
passage_lengthmedium
-22.986
passage_lengthshort
-37.260
modesecs_traditional
4.572
passage_lengthlong:modesecs_traditional
-6.253
passage_lengthmedium:modesecs_traditional
-5.929
passage_lengthshort:modesecs_traditional
-5.419
$`Grade 4`
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula:
time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
mode + passage_length:mode
Data: .
AIC BIC logLik deviance df.resid
58080.42 58158.97 -29029.21 58058.42 9317
Random effects:
Groups Name Std.Dev.
student_id (Intercept) 5.975
passage_id (Intercept) 2.160
Residual 5.027
Number of obs: 9328, groups: student_id, 315; passage_id, 106
Fixed Effects:
(Intercept)
53.552
passage_lengthlong
-11.367
passage_lengthmedium
-29.152
passage_lengthshort
-41.033
modesecs_traditional
3.669
passage_lengthlong:modesecs_traditional
-4.961
passage_lengthmedium:modesecs_traditional
-4.568
passage_lengthshort:modesecs_traditional
-4.412 Model Comparison Test
We conducted a deviance test, comparing the model with main effects only to the model with main effects and their interaction.
$`Grade 2`
Data: .
Models:
.x: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 8 53747 53802 -26865 53731
.y 11 53730 53806 -26854 53708 22.675 3 4.719e-05 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 3`
Data: .
Models:
.x: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 8 64126 64184 -32055 64110
.y 11 64093 64172 -32036 64071 39.308 3 1.493e-08 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
$`Grade 4`
Data: .
Models:
.x: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.x: mode
.y: time ~ 1 + (1 | student_id) + (1 | passage_id) + passage_length +
.y: mode + passage_length:mode
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
.x 8 58104 58161 -29044 58088
.y 11 58080 58159 -29029 58058 29.306 3 1.932e-06 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Our conclusion was that the addition of the passage_length x scoring_method statistically improved the model fit. Thus, our final time included random effects for students and passages, and fixed effects for passage_length, scoring_method, and a passage_length x scoring_method interaction.
Acknowledgments
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A140203 to the University of Oregon. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.