BishtRef
Articles Journals Publishers Authors Subjects Most Cited New Papers Year Stats Open Access
journal article Aug 01, 2016

Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes

Nianbo Dong Wendy M. Reinke ORCID Keith C. Herman ORCID Catherine P. Bradshaw Desiree W. Murray
Evaluation Review Vol. 40 No. 4 pp. 334-377 · SAGE Publications
View at Publisher Save 10.1177/0193841x16671283
Abstract
Background: There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs). Objectives: We calculated the empirical reference values on critical research design parameters associated with statistical power for children’s social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels ( R2). Subjects: Children from kindergarten to Grade 5 in the samples from four large CRTs evaluating the effectiveness of two classroom- and two school-level preventive interventions. Measures: Teacher ratings of students’ social and behavioral outcomes using the Teacher Observation of Classroom Adaptation–Checklist and the Social Competence Scale–Teacher. Research design: Two types of effect size benchmarks were calculated: (1) normative expectations for change and (2) policy-relevant demographic performance gaps. The ICCs and R2 were calculated using two-level hierarchical linear modeling (HLM), where students are nested within schools, and three-level HLM, where students were nested within classrooms, and classrooms were nested within schools. Results and Conclusions: Comprehensive tables of benchmarks and ICC values are provided to inform prevention researchers in interpreting the effect size of interventions and conduct power analyses for designing CRTs of children’s social and behavioral outcomes. The discussion also provides a demonstration for how to use the parameter reference values provided in this article to calculate the sample size for two- and three-level CRTs designs.
Topics

No keywords indexed for this article. Browse by subject →

References
52
[6]
Borenstein M. (2012)
[13]
Corrigan A. (2003). Teacher social competence scale, grade 8/year 9 (Fast Track Project Technical Report). Retrieved from the Fast Track Project website, http://www.fasttrackproject.org
[17]
Gifford-Smith M. (2000)
[22]
Hedges L. V., Rhoads C. (2010). Statistical power analysis in education research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. This report is available on the IES website at http://ies.ed.gov/ncser/
[25]
Johns Hopkins Prevention Intervention Research Center. (2006). The first generation of JHU PIRC preventive intervention trials: Methods and measures. Retrieved December 20, 2015, from http://www.jhsph.edu/prevention/Data/Cohort_1_and_2/Methods_and_Measures
[28]
Kellam S. G. Addiction Science & Clinical Practice (2011)
[31]
Lipsey M. W., Puzio K., Yun C., Hebert M. A., Steinka-Fry K., Cole M. W.…Busick M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncser/
[32]
Murray D. W. (2014)
[33]
National Research Council & Institute of Medicine (2009)
[37]
Raudenbush S. W. (2002)
[39]
Raudenbush S. W., Spybrook J., Congdon R., Liu X., Martinez A., Bloom H., Hill C. (2011). Optimal design plus empirical evidence (Version 3.0). Retrieved January 20, 2015, from http://www.wtgrantfoundation.org
[40]
Reinke W. M. (2014)
[42]
SAS Institute Inc (2013)
[48]
Vanneman A. (2009)

Showing 50 of 52 references

Cited By
49
Promoting Knowledge Accumulation About Intervention Effects: Exploring Strategies for Standardizing Statistical Approaches and Effect Size Reporting

Joseph A. Taylor, Terri Pigott · 2021

Educational Researcher
Metrics
49
Citations
52
References
Details
Published
Aug 01, 2016
Vol/Issue
40(4)
Pages
334-377
License
View
Authors
N
Nianbo Dong

University of Missouri, Columbia, MO, USA

W
Wendy M. Reinke

University of Missouri, Columbia, MO, USA

K
Keith C. Herman

University of Missouri, Columbia, MO, USA

C
Catherine P. Bradshaw

University of Virginia, Charlottesville, VA, USA

D
Desiree W. Murray

University of North Carolina, Chapel Hill, NC, USA

Cite This Article
Nianbo Dong, Wendy M. Reinke, Keith C. Herman, et al. (2016). Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes. Evaluation Review, 40(4), 334-377. https://doi.org/10.1177/0193841x16671283
Related

You May Also Like

Response-Shift Bias

George S. Howard · 1980

373 citations

The Use of Self-Generated Identification Codes in Longitudinal Research

Leo A. Yurek, Joseph Vasey · 2008

87 citations

Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education

Larry V. Hedges, E. C. Hedberg · 2013

74 citations

Achieving SDG-13 in the Era of Conflicts: The Roles of Economic Growth and Government Stability

Jiapeng Dai, Zahoor Ahmed · 2023

39 citations

Covariance Adjustments for the Analysis of Randomized Field Experiments

Richard Berk, Emil Pitkin · 2013

34 citations