What Works in Early Childhood Education Programs?: A Meta-Analysis of Preschool Enhancement Programs

Research Findings: This study uses data from a comprehensive meta-analytic database of early childhood education (ECE) program evaluations published between 1960 and 2007 in the United States to examine the incremental effects of adding enhancement program components to ECE programs on children's cognitive abilities, pre-academic skills, behavioral, health, and socio-emotional outcomes. Preschool enhancement programs include parenting programs, skill-based curricula, and teacher professional development programming. Our findings suggest that the addition of parent programs and skill-based curricula to ECE programs can result in improvements to a range of children's ECE outcomes leading to better school readiness. We found no differences in the impacts of ECE programs with or without additional professional development enhancements. Practice or Policy: Designing fully-developed parent programs by explicitly targeting parents, developing academically focused and skill-based curricula, and providing additional teacher professional development enhancements to existing ECE programs can have a substantial impact on a range of children's ECE outcomes leading to better school readiness. Further research is needed in order to determine what conditions are essential to enhancement program success as well as what conditions have negligible effects on or inhibit children's school readiness.

Early Childhood Education (ECE) can improve children's school readiness. Past studies consistently find that ECE has positive effects on children's cognitive abilities, pre-academic skills, and, in some cases, socio-emotional development (Magnuson, Ruhm, & Waldfogel, [45]; Ramey & Ramey, [60]; Yoshikawa et al., [76]). However, although the field has expanded to serve more children, it is unclear what specific program components constitute the active ingredients for promoting children's early learning and development.

In an effort to improve ECE effectiveness, there have been a variety of approaches taken by scholars. Some have focused on generally improving teaching and classroom management practices. However, others have implemented and evaluated fully developed enhancement programs in the context of evidence-based practice. The programs that have been tested include: parent programs, curricula that focus on specific skills, and professional development for teachers. However, the research that evaluates the added value that comes from these programs is quite limited, and in particular, little attention has been paid to assessing whether the addition of fully developed enhancement programs to existing ECE programs leads to improved cognitive abilities, pre-academic skills, and other positive outcomes among children.

Meta-analytic techniques can be used to quantify a program's impact across studies and systematically identify common patterns of a program's effectiveness. Therefore, we used meta-analytic techniques to examine whether the addition of a fully developed enhancement program (e.g. parent programs, skill-based curricula, professional development) to an existing ECE program will improve children's cognitive (including cognitive abilities and pre-academic skills) and other outcomes related to behavior, health, mental health, and some dimensions of socio-emotional development. To date, our study is the first meta-analysis that systematically studies the impact of adding differing types of fully developed enhancement programs to ECE programs on children's learning outcomes.

Understanding the impacts of adding these enhancement programs can be useful in developing the most effective approach to improving various outcomes for children and leading to better school readiness. Fully developed enhancement programs, also referred to as "add-on programs," differ from those of ad hoc practices because enhancement programs have more targeted goals and have more resources to achieve the goals that are added to existing ECE programs.

ECE programs vary in their practices and occasionally add enhancement services to existing programming. For example, some parenting programs are added to ECE programs in order to involve parents in their children's early learning through workshops, parent meetings, and parent-teacher conferences (Magnuson & Schindler, [44]). Several programs also add specific skill-based curricula that focus on children's language, literacy, and/or math skills to ECE programs and integrate targeted skills in class activities. Skill-based curricula focus more on developing specific skills through intensive and evidence-based curricula compared with "global" curricula, also referred to as the "whole child approach," that have a wider scope (Duncan et al., [26]). Some ECE programs also provide additional professional development for service providers of ECE programs (e.g. teachers) in order to increase providers' knowledge and skills, as well as enhance their teaching and classroom management practices through regular training sessions, mentoring, and/or coaching, going beyond merely providing pre-service or in-service trainings (Yoshikawa et al., [76]). Systematically examining what works in ECE programs is important for finding the most effective strategies that increase children's early learning acquisition and development.

Although many studies have evaluated the overall impact of broader ECE programs, few studies have specifically examined the effectiveness of incorporating specific enhancement programs in ECE settings can improve children's cognitive abilities, pre-academic skills, and other developmental outcomes. Insufficient evidence for the benefits of added enhancement components could be due to the difficulties associated with disentangling the impact of added components from that of ECE programs as a whole (Magnuson & Schindler, [44]). The purpose of this study is, therefore, to use a meta-analytic approach and examine the effect of adding preschool enhancement programs, including parent programs, skill-based curricula (language/literacy or math), and teacher professional development, on children's various developmental outcomes (e.g. cognitive abilities, pre-academic skills, other outcomes), where the comparison control condition is ECE programs without such components. Parent programs, skill-based curricula, and professional development are known to be primary enhancement programs to ECE programs.

Parents have an early and important influence on their children's abilities to learn. Parenting behaviors, such as verbal interactions, responsiveness, and stimulation, have significant impacts on children. According to the developmental theory and empirical findings, the quality and quantity of positive parenting behaviors and parent-child interactions affects children's early language, literacy, and math skills as well as their social and behavioral development, which are all critical components of school readiness and later academic success (Anders et al., [3]; Landry, Smith, Swank, Assel, & Vellet, [40]; McCall, [49]; Ramani & Siegler, [59]). Young children whose parents engage in more accepting, nurturing, and cognitively stimulating behaviors are more likely to exhibit greater growth in their academic skills and have higher self-esteem and academic achievement than children whose parents do not engage in such behaviors (Smith, Perou, & Lesesne, [69]).

Given the importance of parenting for children's development, parenting behaviors are a natural target for early learning interventions that attempt to improve children's learning. Parenting intervention programs often offer guidelines and resources to enhance positive parent-child interactions and promote stimulating parent behaviors that have significant benefits for children. Evaluations of a wide range of parenting interventions have concluded that both parents and children experience positive effects (Breitenstein & Gross, [12]; Brotman et al., [15]; Gross et al., [33]). Therefore, many ECE settings, especially those serving children from disadvantaged or vulnerable family backgrounds, provide some parent-related programs.

A small number of meta-analysis studies have examined the benefits of parent programs that are embedded in ECE settings. They have produced mixed results about whether these programs are effective at improving children's cognitive abilities and pre-academic skills. Two recent studies, which used a subset of studies from the same meta-analysis database used for this study, systematically reviewed how children's early academic skills are impacted by the parent programs in ECE settings. One study examined the benefits of preschool services and other social services that included parenting programs and unexpectedly found that they had significantly smaller effects on children's cognitive development than programs which provided only preschool services (Camilli, Vargas, Ryan, & Barnett, [19]). The second study analyzed the impacts of ECE programs that embedded parenting program components, including parenting practices such as modeling and opportunities to practice stimulating behaviors (Grindal et al., [32]). Examples of these parenting components include parent-child activities during home visiting or in classroom settings, individual contact with parents to support parent-child interactions, and engagement between the parents and the school. The study found no difference in cognitive ability or pre-academic skills between children who attended preschools that embedded parenting programs and children who attended preschools that did not (Grindal et al., [32]). However, the broader ECE programs did have positive effects on children's outcomes (Grindal et al., [32]). Among the ECE programs that provided parenting education, those that involved one or more home visits a month yielded effect sizes for cognitive outcomes that were significantly larger than programs that provided less frequent home visits (Grindal et al., [32]). However, the two meta-analysis studies examined parenting practices that were part of the preschools themselves and not distinct parent enhancement programs. Further, they did not consider the differences between parenting practices and fully developed parenting programs.

The content and structure of parent programs can vary widely. Some programs promote a set of parenting practices that encourage parent-child interactions, whereas others are fully developed, curriculum-based parental programs that identify more targeted goals, employ professional staff members to achieve those goals, and have more resources to provide services as intended (Magnuson & Schindler, [44]). Thus, the results of fully developed parent programs may differ from those that use ad hoc practices.

Intervention studies that examine the benefits of adding parent enhancements to ECE settings show some promising results. For example, the REDI (Research-based Developmentally Informed) Parent (REDI-P) program provides additional home visits to parents of preschool children in Head Start. It is designed to support parents use learning activities and games at home to enhance children's school readiness beyond the classroom. An evaluation of the REDI-P program showed significant improvements in language and literacy skills (d =.25-.28), social-emotional adjustment (d =.28-.29), as well as parental support for learning (d =.27-.28) compared to the control group (Bierman, Welsh, Heinrichs, Nix, & Mathis, [8]). Another parent enhancement program, Parent Corps, provides a series of 13 two-hour group sessions for parents and children facilitated by preschool teachers and mental health professionals, to help develop young children's foundational skills for learning. An evaluation study of Parent Corps identified improvements in parenting practices (d =.50) and child behavior problems (d =.56) (Brotman et al., [14]). Given the consistent positive findings of well-designed parent enhancement programs, this study focuses on parent enhancement programs that are evidence-based and fully developed, and examines the effect of adding fully developed parent programs to ECE settings on children's learning outcomes.

Curricula may boost children's development by guiding teachers' instruction in classroom activities, enabling teachers to rely on structured learning activities rather than their own preparations, and allowing teachers to use novel materials in the classroom. Most ECE programs use some form of curricula to support children's learning and stimulate interaction with other children and teachers. Whereas the whole child approach, also referred to as "global" curricula, emphasizes children's overall development by encouraging children to interact independently with materials and other children in the classroom setting (DeVries & Kohlberg, [25]; Piaget, [52]), some ECE programs use curricula that is more content specific and tailored to developing specific skills in children, such as literacy/language and/or math skills (Duncan et al., [26]; Yoshikawa et al., [76]). Compared with programs that take the whole child approach, skill-based curricula provide more explicit academic instruction and focuses a portion of the day on developing specific skills (Duncan et al., [26]).

Findings from the Preschool Curriculum Evaluation Research (PCER) funded by the Institute of Education Sciences (IES) illustrate mixed evidence regarding the effectiveness of the treatment curricula compared to the counterfactual control condition (commonly teacher-developed curricula) on children's school readiness. Each research team collaborated with teachers and implemented different preschool curricula to improve children's school readiness, including early reading, language, and mathematical skills, phonological awareness, and behaviors and randomly assigned classrooms with treatment curricula and compared classrooms with teacher-developed curricula (Preschool Curriculum Evaluation Research Consortium [PCER], [57]). Results indicated that 13 curricula had no impact on reading skills or phonological awareness, whereas compared to teacher-developed curricula, the DLM Early Childhood Express with Open Court Reading Pre-K curriculum showed positive impacts on children's phonological awareness and reading and language skills (PCER, [57]). Regarding children's mathematical skills, only Pre-K Mathematics with DLM Early Childhood Express Math software curriculum had positive effects compared to teacher-developed curricula, while the other 13 curricula had no statistically significant impact on children's mathematical skills (PCER, [57]). No curriculum had a statistically significant impact on children's behavioral outcomes compared to the control condition curricula (PCER, [57]).

The re-analysis of the PCER study (Duncan et al., [26]), however, suggests some promising results regarding the effect of skill-based instruction on children's school readiness compared with programs that take the whole child approach. The re-analysis study compared the four types of preschool curricula: literacy-focused, mathematics-focused, whole-child (e.g. Creative Curriculum, HighScope Curriculum), and locally-developed curricula (developed by teacher or school). It found that children in classrooms that were randomly assigned to a literacy curriculum had a higher literacy composite score (e.g. PPVT, WJ Letter Word, Spelling) than children assigned to Creative/HighScope curriculum and a higher math composite score (e.g. WJ Applied Problems, CMAA) than children assigned to locally-developed curricula (Duncan et al., [26]). Children in classrooms randomly assigned to math curriculum had a higher math composite score than children in classrooms assigned to Creative/HighScope Curriculum (Duncan et al., [26]). There were no statistically significant differences in children's school readiness between those assigned to Creative/HighScope and locally-developed curricula (Duncan et al., [26]).

Consistent with the findings from the PCER re-analysis study, small-scale single curriculum studies also show the positive effects of specific skill-based curricula. For example, the Exemplary Model of Early Reading Growth and Excellence (EMERGE) program, which is designed to improve children's early literacy skills for children from low-income families, provides curriculum including interactive shared book reading and classroom activities focusing on literacy development. The evaluation of the EMERGE identified large improvements in children's literacy and language at the end of preschool (Gettinger & Stoiber, [30]). An evaluation of the Building Blocks curriculum, which is a preschool mathematics curriculum designed to provide daily whole- and small-group activities and materials for integrating mathematics in the classroom, also suggests that research-based software and print curricula on mathematics improved children's mathematical knowledge of numbers and geometry (Clements & Sarama, [21]).

Given the significant positive impacts of skill-based curriculum implementation on school readiness, this study focuses on the curricula that specifically target either or both literacy and math, and examine the added impact of providing content specific curricula to ECE programs. Because not every ECE program provides skill-focused approach and academically focused curricula to support children's early learning and development, examining the added impact of providing these curricula compared to programs without skill-based curricula may be important for early childhood educators in identifying areas that necessitate further attention.

Professional development is considered to be important in ECE programs, given the critical role of care providers in child development. Care providers, who can also be educators, have a significant impact on children's development because they provide direct education to children. Besides parents, teachers play an important role in shaping children's early learning outcomes. Because improvements in care providers' knowledge and/or practices are expected to improve children's development (Markussen-Brown et al., [46]; Powell, Diamond, Burchinal, & Koehler, [56]), most ECE programs acknowledge the importance of professional development and provides some form of training to educators.

Professional development generally includes facilitating teacher-child interactions and helping educators use novel materials through formal courses, workshops, conferences, mentoring and/or coaching (Lawless & Pellegrino, [41]; Powell et al., [56]). While most ECE programs provide at least some form of pre-service or in-service training through didactic workshops, some programs provide additional professional development programs that have more targeted goals and integrate evidence-based strategies, tailed to improving children's skills, through in-classroom mentoring or coaching (Yoshikawa et al., [76]). Past studies, however, suggest mixed evidence regarding whether professional development has a positive effect on care providers and/or children. The findings from Making the Most of Classroom Interactions (MMCI), designed to provide teachers with face-to-face professional development through regular meetings, suggest that teachers participating in MMCI had higher scores in emotional support, instructional support, and classroom organization than teachers who did not participate in MMCI (Early, Maxwell, Ponder, & Pan, [27]). In addition, My Teaching Partner (MTP), which is designed to provide teachers with one-on-one coaching on classroom instruction and practices, had positive effects on teacher-child interactions (Pianta, Mashburn, Downer, Hamre, & Justice, [53]) and children's language and literacy skill development (Mashburn, Downer, Hamre, Justice, & Pianta, [48]). However, other studies found little or no effects on children's outcomes (Buysse, Castro, & Peisner-Feinberg, [17]; Cabell et al., [18]).

Several studies have systematically reviewed literature on teacher professional development to examine whether professional development is helpful in improving teachers and children's outcomes. Zaslow, Tout, Halle, Whittaker, and Lavelle ([77]) found mixed evidence regarding the impact of teacher professional development programs on early educators' knowledge or practices or on child outcomes. Programs having more targeted goals and intense training that is matched to the program content generally had positive impacts on educators' practices and child outcomes. Markussen-Brown et al. ([46]) estimated the effect of language- and literacy-focused professional development and found positive effects on educator-child interactions, receptive vocabulary, phonological awareness, and alphabet knowledge but found no effect on educators' knowledge (Markussen-Brown et al., [46]). The mixed nature of evidence from past studies may arise because the majority of studies are of a small scale and the content, format, and structure of professional development programs vary widely. Some professional development programs may use formal courses and workshops, while others may use coaches or study groups.

Building on the findings from Markussen-Brown et al. ([46]) and Zaslow et al. ([77]), this study focuses on exploring whether adding evidence-based professional development, rather than just simply providing pre-service or in-service training through didactic workshops, improves children's various developmental outcomes not limited to language. Additional professional development layered on the top of ECE programs has more of a structured format and integrates evidence-based strategies to train teachers as intended. In order to ensure that services are provided as intended, programs tend to administer regular training sessions, supplemented with mentoring or coaching, and monitor program implementations on a regular basis.

No recent review has encompassed add-on professional development studies in areas other than language; thus, this study will add to the literature through its examination of how various outcomes in children can be positively impacted through the addition of teacher professional development. Examining the benefit of adding teacher professional development programs may be important because it allows researchers to distinguish the impact of added components from that of ECE programs as a whole.

The advantage of a meta-analysis is that it combines and identifies common patterns in findings from a wide range of studies on a particular topic. This meta-analysis study focuses on estimating the effects of implementing enhancement programs in ECE settings on a range of children's developmental outcomes, including their cognitive abilities, pre-academic skills, health, mental health, and behaviors as well as some dimensions of socio-emotional developmental outcomes. Specifically, this study addresses the following questions:

In this study, we analyzed a subset of studies from a large meta-analytic database of ECE studies developed by the National Forum on Early Childhood Policy and Programs. The Forum used a multi-step data collection and screening process to determine which studies to include in the database. The process began by getting complete information of studies evaluating ECE programs for 3- to 5-year-old children and of studies in another meta-analytic database complied by Abt Associates, Inc. and the National Institute for Early Education Research (NIEER) (Jacob, Creps & Boulay, [38]; Camilli et al., [19]; Layzer, Goodson, Bernstein, & Price, [42]). Next, the Forum's database updated Abt's list to include other articles published in recent years that were not included, and also added programs for children up to 3 years of age. Next, the Forum conducted keyword searches in the ERIC, PsychINFO, EconLit, and Dissertation Abstract databases and manually searched the websites of policy institutions (e.g., RAND, Mathematica, NIEER) and state and federal agencies (e.g., the United States Department of Health and Human Services). Project staff also collected references mentioned in included studies and other important literature reviews. A total of 10,309 reports were identified as potential candidates for inclusion in the overall database.

Seven criteria were applied by the Forum to determine which reports should be included in the larger database: 1) the intervention studied provided services for children, parents, teachers/caregivers, or communities/neighborhoods; 2) the study had a comparison group (alternative treatment or no treatment control group); 3) the ages of children during the intervention ranged from prenatal to 5 years old; 4) the research had at least 10 children in each comparison group; 5) the intervention was not intended for children with learning disabilities or diagnosed behavioral, emotional, or medical disorders or; 6) the intervention did not involve pharmacological agents, medical procedures, or health-related products; and 7) the study was conducted in the United States and published after 1960 and before 2007. These above criteria are more rigorous than those applied by Abt Associates and NIEER because the Forum excluded evaluation studies with no comparison group as well as studies which did not examined the baseline equivalence of treatment and control group. The vast majority (91%) of the studies were excluded because they did not meet at least one of these basic inclusion criterion. The Forum's final database included 277 studies on ECE programs for children up to 5 years of age.

The selected 277 studies were coded by a team of nine doctoral-level graduate research assistants at four universities (Harvard University, New York University, University of California–Irvine, and University of Wisconsin–Madison). The coders were trained over three to six months on how to document the study design, programs, sample characteristics, and other information needed to compute effect sizes. Before conducting coding independently, the coders were paired with experienced coders during multiple rounds of practice coding and had to pass a reliability test (percent agreement) of 100% for effect size and 80% for all other study information with a master coder. The range of interrater reliability for all study data was 0.87–0.96 (M = 0.87). Coding questions were resolved during weekly research team conference calls between coders and principal investigators, and decisions were recorded in a codebook and meeting notes. Before the Forum's final database was constructed, the data entry was checked through systematic data exploration as well as cleaning, testing outliers and examining missing values.

The database of the National Forum on Early Childhood Policy and Program consisted of three-level data: studies, contrasts, and effect size. A study was defined as an independent evaluation of ECE programs. Each study had a number of contrasts, defined as comparison groups with different treatment conditions within a single study. There were two types of contrasts: treatment vs. control contrasts and treatment vs. alternative treatment contrasts. Effect sizes (Hedges' g) for treatment impacts were calculated using Comprehensive Meta-analysis software (CMA; Borenstein, Hedges, Higgins, & Rothstein, [10]). Hedges' g adjusted the standardized mean differences (Cohen's d) to account for bias mostly due to small sample sizes (Schindler et al., [64]).

In the present study, we used a subset of the Forum's database and additional inclusion criteria were applied to studies: 1) a study had to measure children's developmental outcomes, such as cognitive abilities and pre-academic skills, behavioral and socio-emotional outcomes, and mental and physical health; and 2) a treatment with a specific enhancement program (e.g., parenting programs, skill-based curricula, professional development) was compared to an ECE program without such a component.

Of the 124 studies reviewed (that contained any type of enhancement program), 10 studies included parenting enhancements (134 effect sizes nested in 11 contrasts), 21 studies included curricular enhancements (176 effect sizes nested in 32 contrasts), and 7 studies included professional development enhancements (43 effect sizes nested in 9 contrasts). The types of enhancement programs were not mutually exclusive, resulting in 38 studies. However, it is important to note that there was only one contrast that had both curricular and professional development enhancements, and we included this separately in the analyses in order to examine the average treatment effects in each enhancement program on children's outcomes.

Regarding curricular enhancements, this study divided curricular enhancements into three categories: 1) language/literacy specific, 2) math specific, and 3) other content specific comprehensive curricula. Language/literature specific curricula had 150 effect sizes nested in 25 contrasts, math specific curricula had 12 effect sizes nested in 5 contrasts, and other curricula had 14 effect sizes nested in 2 contrasts. Other curricula is research-based content specific curricula that focused on various aspects of child development, including language/literacy, math, science, fine arts, health, and social development. Because the number of effect sizes was very small in math specific (n = 12) and other curricula (n = 14) compared to language specific curricula, we combined math specific and other curricula when analyzing the average treatment effect with (see Appendix A for more information on add-on enhancement programs).

Effect sizes were measured in three domains, and these domains were combined in some analyses. Cognitive abilities covered were measures of the theory of mind, attention, intelligence quotient (IQ), vocabulary, task persistence, and syllabic segmentation, such as elision and rhyming. Pre-academic skills considered included reading, math, letter recognition, numeracy (other than conservation of number), and other achievement tests. Behavioral, health, and some dimensions of socio-emotional outcomes included internalizing, externalizing, aggressive/antisocial behavior, withdrawal, emotional regulation, relational skills, mental and physical health, and all other developmental outcomes. Table 1 shows descriptive statistics, including the weighted and unweighted average effect sizes of enhancement programs.

Table 1. Descriptive statistics for all variables by enhancement programs.

1 Note: 1) Effect sizes were weighted by the inverse variance weight of each effect size multiplied by the inverse of the number of effect sizes within a program to give more weights to precise estimates and less weights to effect sizes in contrasts with a large number of outcomes; 2) Types of add-on enhancement programs and contrasts are not mutually exclusive.

Parent enhancements were designed to teach parents how to better support their children's early learning by providing stimulating interactions during daily routines and playtime at home. ECE parent programs were delivered in a variety of formats including parent training, group meetings, family classes, and/or provided with at-home instructional materials to facilitate children's early learning process. Each parent enhancement program generally had a specific target of intervention, such as children's cognitive development, phonemic awareness, and mathematics. Most parent enhancements lasted less than 1 year.

For curricular enhancements, each program provided content specific curricula to enhance specific early learning skills such as children's literacy, language development, or math. Language specific curricula were designed to provide teachers with structured guidelines to promote literacy/language skills in classroom activities (e.g. role play, reading books, etc.) and materials. Math specific curricula were designed to integrate mathematics activities in classroom activities, computer software, games, and other materials. Our category of other curricula included programs that have academically focused and research-based curricula focusing on literacy/language and/or math, combined with other contents such as science, arts, and physical and social development. We combined math specific and other curricula in our model because the number of effect sizes was too small.

Professional development enhancements were designed to provide additional training, workshops, mentoring, and/or coaching for teachers in ECE programs. Most professional development enhancement programs targeted teaching practices that would improve children's language and literacy skills through training, workshops as well as mentoring or coaching. Teachers were trained to apply effective teaching strategies in classroom activities, including book readings and conversation strategies. Structures and formats of professional development programs were research-based and provided regular training sessions with supplemental mentoring or coaching.

It was uncommon for enhancements to have more than one type of added component in each contrast. There was only one contrast that provided the combination of a literacy-focused curriculum and coaching. We included this contrast in both curricular enhancements and professional development enhancements in the analyses in order to examine the effects of the average treatment in each enhancement program on children's outcomes (see Appendix A for more information on add-on enhancement programs). All of the enhancements were provided when children were ages 3 to 5-years old.

In meta-analysis, it is important to include measures of differences in the study design or measures that may be systematically associated with effect sizes. For this reason, we included several covariates that capture differing study designs and effect size characteristics. Based on the ECE literature and prior meta-analytic studies, we controlled for several aspects of the study design and program characteristics in the analyses (e.g., Shager et al., [65]). Regarding the study design, we created a study quality index based on whether a study used random assignment (1/0), whether the coders observed any evidence of systematic bias in a study (1/0), and whether there was a high attrition rate (1/0). A higher value for the index indicated a low study quality. The weighted average of study quality index was 1.0 (SD = 0.7) for parent enhancements, 1.1 (SD = 0.6) for curricular enhancement, and 1.7 (SD = 0.6) for professional development enhancements. We also controlled for whether the measures were collected at the end of the program (1/0) or at a later follow-up wave of data collection. For parent enhancements, 25% were measured at follow-up and 75% were measured during or at the end of the treatment. For curricular enhancements, all programs were measured during or at the end of the treatment. For professional development enhancements, 7% were measured at follow-up and 93% were measured during or at the end of the treatment. Table 1 displays the summary statistics (mean, standard deviation, and sample sizes).

Using standard meta-analytic techniques, we estimated the average effect sizes (Hedges' g) for children's cognitive abilities and pre-academic skills as well as other outcomes, associated with participating in the program treatment group. Multi-level regression models were employed to handle the nested data structure (effect sizes within contrasts). Although the data has three levels (effect sizes, contrasts, and study), we used two-level models because most of the studies included only one contrast, and thus, the data could not support a three-level analysis. In addition, the effect sizes tend to be similar within contrasts than within studies in the data used in the analyses.

Our models with nested equations (Raudenbush & Bryk, [61]) are as follows: The level 1 model (effect size level) is:

(1)

Graph

$E{S}_{ij}={\pi }_{0j}+{\pi }_{1j}{x}_{1ij}+...+{\pi }_{kj}{x}_{kij}+e_{ij,}$

where effect size i in contrast j is modeled as a function of the intercept (π0j), which represents the average (covariate adjusted) effect size for all contrasts; of a set of estimated effects of k independent variables measured at the effect size level (π1jx1ij + ... + πkjxkij); and of a within-contrast error term (eij).

The level 2 equation (contrast level) models the intercept as a function of the grand mean effect size (β00), p independent variables measured at the contrast level (β01x1j + ... + β0pxpj), and a between-contrast random error term (u0j):

(2)

Graph

${\pi }_{0j}={\beta }_{00}+{\beta }_{01}{x}_{1j}+...+{\beta }_{0p}{x}_{pj+}{u}_{0j}$

We used the XTMIXED command in Stata 14.1 to conduct all the analyses. Weights were designed to give more weight to precise estimates and to reduce the weight of contrasts with many measured outcomes. This was done by constructing weights that multiplied the inverse variance (which gave more weight to precise estimates) by 1/N, where N reflects the number of effects sizes within a contrast (thus studies with a large number of outcomes were not given undue weight).

There were no missing data on covariates, including study design, bias, attrition, and timing of the measure in our analytic sample. However, 12 effect sizes were missing in parenting enhancements, 27 effect sizes were missing in skill-based curricula enhancements, and 2 effect sizes were missing in professional development enhancements. We used missing data imputation in Stata to impute the missing effect sizes, generated 20 imputed datasets, and conducted a sensitivity analysis that included imputed missing effect sizes.

One concern about meta-analytic techniques is that findings from unpublished studies may have been difficult to find and that results from unpublished studies are systematically different from published studies. To test for publication bias, we used a funnel plot, which presented the average effect sizes for each contrast and the standard error, and conducted Egger's test to check for the asymmetry of the funnel plot (see Figure 1; Sterne & Egger, [71]). The Egger's test suggested a low likelihood of publication bias, (p =.45). Although the pattern was not normally distributed as might be expected and showed some indication of positive outliers and a comparative lack of negative outliers, Egger's regression test yielded no indication of significant asymmetry. We also presented a funnel plot by enhancement programs to examine which types of enhancement were exceptionally located on the positive side of the funnel plot (see Figure 1). Among four contrasts with effect sizes larger than 1, three contrasts were skill-based curricula and one contrast was an evaluation of a parent enhancement program. To consider whether our results were sensitive to these positive outliers, we estimated models in which they were excluded.

PHOTO (COLOR): Figure 1. Funnel plot of average effect sizes for each contrasts and standard error (N = 51 contrasts, 350 effect sizes). Note: Funnel plot shows low likelihood of publication bias (p =.45). Although pattern was not normally distributed as might be expected and showed some indication of positive outliers and a comparative lack of negative outliers, Egger's regression test yielded no indication of significant asymmetry. For robustness check, we excluded contrasts with large average effect sizes, which were exceptionally on positive side of the funnel plot. Four contrasts had average effect sizes larger than 1 (see Figure 2). Estimates, excluding four contrasts, were consistent with results presented in Table 2. Although the average effect sizes became smaller in magnitude, the significance levels were highly consistent.

PHOTO (COLOR): Figure 2. Forest plot and Q statistics to test for heterogeneity in the effect sizes by contrasts (N = 51 contrasts, 350 effect sizes). Note: The I-squared index shows that 95.3% of variations in effect size are due to heterogeneity (between study variability) rather than chance (sampling error), concluding high heterogeneity in the effect sizes by contrasts.

In order to explain variation in effect sizes, we need significant heterogeneity in effect sizes. To consider if there was sufficient variation to be explained, we used the Q test proposed by Cochran ([22]). Q statistics tests for statistically significant between-study variation. Although Q statistics detect heterogeneity, they do not measure the degree of heterogeneity (Heudo-Medina, Sanchez-Meca, Marin-Martinex, & Botellam, [35]). To quantify the degree of heterogeneity, Higgins and Thompson ([36]) proposed the I-squared index, which divides the difference between the results of the Q test and its degree of freedom by the Q value itself and multiplies it by 100.

Graph

$I^2=\left(\frac{Q-df}{Q}\right)\times 100\mathrm{\%}$

The I-squared index shows the percentage of the total variability in a set of effect sizes due to true heterogeneity (between-studies variability). A large percentage in the I-squared index indicates high heterogeneity. The Q value indicated statistically significant between-contrast variation (Q = 1064.47, p <.001). The forest plot in Figure 2 confirmed the high degree of heterogeneity. The I-squared value indicated that 95.3% of variations in effect size were due to heterogeneity (between study variability) rather than chance (sampling error).

Our first goal was to estimate the average effect size impacts on children's outcomes for three different types of enhancement programs. First, we considered the average effect sizes for parent enhancements on children's overall performance, cognitive abilities, pre-academic skills, and behavioral, health, and some dimensions of socio-emotional outcomes. Table 2 presents the multi-level regression models for all outcomes, as well as separate models for children's cognitive abilities, pre-academic skills, and behavioral, health, and some dimensions of socio-emotional outcomes. Controlling for the study quality index and the timing of the measurement, the average treatment effect of adding parent enhancements to ECE programs were significant and positive for all outcomes (ES = 0.28 SD), cognitive abilities (ES = 0.37 SD), and behavioral, health, and socio-emotional outcomes (ES = 0.55 SD). Parent enhancements were not associated with any impact on children's pre-academic skills.

Table 2. Average treatment effect of enhancement programs by outcome domain (with controls) (N = 51 contrasts, 350 effect sizes).

2 Note: +p < 0.10, *p <.05, **p < 0.01, ***p < 0.001.; Standard errors are in parentheses; Models control for the Study Quality Index and type of post-test.; Because the number of effect sizes was too small in math-specific and all other curricula, we combined in the models.; All other curricula include research-based curricular enhancements focusing on language/literacy, math, science, arts, and socio-emotional development.; The types of enhancement programs were not mutually exclusive. However, there was only 1 contrast that had both curricular and professional development enhancement and we included this separately in the analyses to examine the average treatment effects in each enhancement program on children's outcomes.; For sensitivity test, we sliced cognitive outcome to smaller categories (e.g. language/verbal, phonological awareness, vocabulary development) and found positive and significant treatment effects (language: ES = 0.46, SE = 0.12, p <.001; vocabulary: ES = 0.47, SE = 0.13, p <.001; phonological awareness: ES = 0.51, SE = 0.31, p <.01). However, the number of effect size was too small to be analyzed separately and brought concern about statistical power. Thus, we only reported the combined measure.

Our second question aimed to estimate the average effect size of the addition of skill-based curricula to an ECE setting on children's outcomes. Controlling for the study quality index and the timing of the measure, the average treatment effect of adding skill-based curricula to ECE had positive effects on overall performance (ES = 0.70 SD), cognitive abilities (ES = 0.55 SD), and pre-academic skills (ES = 0.51 SD). The average treatment effect of adding skill-based curricula to ECE had significant but small positive effects on behavioral, health, and socio-emotional outcomes (ES = 0.06 SD). However, there were only 4 effect sizes in 2 contrasts and these contrasts differ on the quality index, and the 4 effect sizes were highly correlated with the quality index (Point-Biserial Correlation = 0.83), such that it is hard to feel confident that the estimate with the covariates included is meaningful, rather than spurious, given the confounds and small number of effect sizes.

Among skill-based curricula, literacy/language enhancements had positive effects on overall performance (ES = 0.66 SD), cognitive abilities (ES = 0.59 SD), and pre-academic skills (ES = 0.36 SD). Math specific and other curricula had positive effects on overall performance (ES = 0.93 SD) and pre-academic skills (ES = 0.84 SD) but had no statistically significant effects on cognitive abilities. This is the same analysis as described above for the all curricular enhancements, meaning that all the included curricula with measures of health behavior, and socio-emotional outcomes were language and literacy curricula studies. Thus, the estimates were exactly the same, as our concern about interpreting the estimate as meaningful is still relevant.

Our third question aimed to estimate the average effect size of teacher professional development enhancements for children's outcomes. Controlling for the study quality index and the timing of the measure, the average treatment effect of adding intensive professional development to ECE programs had small but negative effects on pre-academic skills (ES = −0.18 SD). Associations with overall performance and cognitive abilities were not statistically significant. There was an insufficient number of effect sizes and contrasts to measure the average treatment effect of professional development enhancements on behavioral, health, and socio-emotional outcomes (just 4 effect sizes in 1 contrast).

To check the robustness of our results to alternative specifications, we excluded contrasts with large average effect sizes, which were exceptionally located on the positive side of the funnel plot (see Figure 1). Four contrasts had average effect sizes larger than 1 (see Figure 2). Estimates, from analyses excluding four contrasts with large effect sizes, were consistent with the results presented in Table 2. Although the average effect sizes became smaller in magnitude, the estimates were still significantly different from zero. We also checked the robustness of the results due to missing effects sizes within contrasts. As described in our analytic strategy, we conducted multiple imputation methods to impute missing effect sizes. Results that included the imputed effect sizes were consistent with the results presented in Table 2.

As public Early Childhood Education (ECE) program investments have increased and enrollment has expanded, a question of growing importance is how to design ECE programs in order to maximize their effectiveness. However, it is unclear what specific program components are beneficial for promoting children's early learning and development. This study provides the evidence that several approaches to improving ECE impacts can be effective. By using data from a comprehensive meta-analytic database of ECE program evaluations published in the United States between 1960 and 2007, we examined the addition of three different types of preschool enhancement programs to existing ECE programs in order to determine if those programs resulted in improvements to children's cognitive abilities, pre-academic skills, and behavioral, health, and some dimensions of socio-emotional outcomes. Preschool enhancement programs included fully developed parent programs, skill-based curricula, and additional professional development for teachers. Our findings indicate that adding parent enhancements and skill-based curricula to ECE programs has a positive impact on children's school readiness.

Fully developed parent programs differ from ad hoc parent practices and services because they directly focus on improving children's specific skills or parenting behaviors, and these well-developed programs may have more available resources for providing services as intended (Magnuson & Schindler, [44]). We found that adding fully developed parent programs to ECE was associated with large improvements to children's behavioral, health, and some dimensions of socio-emotional outcomes, as well as modest improvements to cognitive abilities and overall outcomes in general. However, adding fully developed parent programs to ECE was not associated with improved pre-academic skills compared to ECE without such programs. This result may imply that children develop pre-academic skills primarily through in-classroom activities in which they learn from teachers rather than through parent programs. Designing parent programs that are integrated with academic skill development may be useful for improving children's pre-academic skills, especially around transition into kindergarten (Bierman, Domitrovich, et al., [7]). Nevertheless, these findings point to the potential of parent programs to improve key learning outcomes for children attending ECE settings.

In our data, adding skill-based curricula to ECE programs, especially literacy/language-specific curricula, was associated with large improvements in children's cognitive abilities, pre-academic skills, and overall outcomes. Math specific and all other curricula were associated with large improvements in children's pre-academic skills and overall outcomes but not cognitive abilities. Although most ECE programs use some form of curricula to promote children's early learning, not every program has evidence-based and academically focused curricula. Skill-based curricula tend to be evidence-based and target specific skill development, whereas "global" curricula are more general and take a broader approach to supporting children's broad developmental outcomes. Our findings were consistent with a recent evaluation study by Duncan and colleagues (Duncan et al., [26]) on the effectiveness of content-rich curricula focused on promoting specific skills compared to widely-used "global" curricula. These consistent findings suggest that early childhood educators and policy-makers should prioritize moving to implement academically focused, skill-based curricula instead of relying on "global" curricula to support children's early learning and development. However, we caution that skill-based is not at all synonymous with didactic teacher developmentally inappropriate instruction. Indeed, well designed curricula that include developmentally appropriate activities that are fun and able to scaffold skills development will be beneficial for children's early learning and development.

Finally, we also examined the benefits of providing additional professional development for teachers compared to ECE programs without additional professional development. Although most ECE programs provide some forms of pre- or in-service training to their teachers, other programs provide additional professional development programs geared toward improving children's skills (in our data often language and literacy skills) through in-classroom mentoring or coaching. Unexpectedly, the findings led to negative significant effects for pre-academic skills and null effects for cognitive abilities or overall outcomes. The number of effect sizes and contrasts were too small to measure effects on behavioral, health, and socio-emotional outcomes. This unexpected result suggests that there is more to learn about the role that professional development may play in improving children's outcomes in ECE settings. Indeed, recent evidence suggests the benefits of in-classroom coaching and mentoring in support of instruction and content specific curricula on children's targeted outcomes (Yoshikawa et al., [76]). Yet, this cautions that more work is needed to understand how to ensure that these efforts bring about the intended outcomes.

It is worth noting that in the past 50 years of research on ECE, far more attention has been given to studying variations in curricula than parenting programs or professional development, although the curricula evaluations can include professional development. Specifically, curricula studies were four times more likely to be conducted than professional development studies. Given the important role teachers play in children's early learning, more research is needed to evaluate the impact of intensive professional development on children's school readiness.

Although our study found promising results with a comprehensive meta-analytic database of ECE program evaluations, there are several notable limitations. First, the findings from this study are unable to specify the mechanisms of how preschool enhancement programs are beneficial to children's early learning and development. Although ECE programs with preschool enhancements added can be more successful than ECE programs without such components, more attention to specific details of the enhancement programs is needed in order to better understand the necessary conditions for program success or ineffectiveness. In addition, these findings cannot be used to infer causality. There may be unmeasured program characteristics that cause a bias leading to incorrect results. Moreover, we could not take into account implementation fidelity because most of the studies did not provide sufficient evidence to judge quality of the implementation.

Despite several study limitations, our findings have policy and practice implications. Federal and state ECE programs should consider designing fully-developed parent programs by explicitly targeting parents, developing academically focused and skill-based curricula, and providing additional teacher professional development enhancements to existing ECE programs. When these enhancements are well-developed and geared toward developing specific skills, ECE programs can have a substantial impact on a range of children's ECE outcomes leading to better school readiness.

We are grateful to the following funders of the National Forum on Early Childhood Policy and Programs: the Birth to Five Policy Alliance, the Buffett Early Childhood Fund, Casey Family Programs, the McCormick Tribune Foundation, the Norlien Foundation, Harvard University, and an Anonymous Donor. We are also grateful to the Institute of Education Sciences, US Department of Education for supporting this research (#R305A110035), and to Abt Associates, Inc. and the National Institute for Early Education Research for making their data available to us.

No potential conflict of interest was reported by the authors.