Multimodel Ensemble ENSO Prediction with CCSM and CFS

2908

MONTHLY WEATHER REVIEW VOLUME 137

Multimodel Ensemble ENSO Prediction with CCSM and CFS

BEN P. KITMAN

Rosenstiel School for Marine and Atmospheric Science, University of Miami, Miami, Florida, and Center for Ocean-Land-Atmosphere Studies, Calverton, Maryland

DUGHONG MIN

Rosenstiel School for Marine and Atmospheric Science, University of Miami, Miami, Florida

(Manuscript received 9 June 2008, in final form 1 March 2009)

ABSTRACT

Results are described from a large sample of coupled ocean-atmosphere retrospective forecasts during 1982-98. The prediction system is based on the National Center for Atmospheric Research (NCAR) Community Climate System Model, version 3 (CCSM3.0), and a state-of-the-art ocean data assimilation system made available by the National Oceanic and Atmospheric Administration (NOAA) Geophysical Fluid Dynamics Laboratory (GFDL). The retrospective forecasts are initialized in January, April, July, and November of each year, and ensembles of 6 forecasts are run for each initial month, yielding a total of 408 1-yr predictions. In generating the ensemble members, perturbations are added to the atmospheric initial state only. The skill of the prediction system is analyzed from both a deterministic and a probabilistic perspective, it is then compared to the operational NOAA Climate Forecast System (CFS), and the forecasts are combined with CFS to produce a multimodel prediction system. While the skill scores for each model are highly dependent on lead time and initialization month, the overall level of skill of the individual models is quite comparable. The multimodel combination (i.e., the unweighted average of the forecast), while not always the most skillful, is generally as skillful as the best model, using either deterministic or probabilistic skill metrics.

1. Introduction

The ability to predict the seasonal variations of the earth's tropical climate dramatically improved from the early 1980s to the late 1990s. This period was bracketed by two of the largest El Ni�o events on record: the 1982/83 event, whose occurrence was unknown until after it was over; and the 1997/98 event, which was predicted up to 6 months in advance. This improvement was due to the convergence of many factors including a concerted international effort to observe, understand, and predict tropical climate variability, the application of theoretical understanding of coupled ocean-atmosphere dynamics, and the development and application of models that accurately simulated the observed variability. The improvement led to considerable optimism regarding our ability to predict seasonal climate variations in general

and El Ni�o-Southern Oscillation (ENSO) events in particular.

After the late 1990s, our ability to predict tropical climate fluctuations reached a plateau with only modest subsequent improvement in skill. Arguably, there were substantial qualitative forecasting successes�almost all the models predicted a warm event during the boreal winter of 1997/98, one to two seasons in advance. Despite these successes, there have also been some striking quantitative failures. For example, according to Barnston et al. (1999) and Landsea and Knaff (2000), the performance of many different prediction systems during the 1997-99 ENSO episodes was mixed. None of the models predicted the early onset or the amplitude of that event and many of the dynamical (i.e., CGCM) forecast systems had difficulty capturing the demise of the warm event and the development of cold anomalies that persisted through 2001. The statistical models did not suffer from this problem. Many models failed to predict the three consecutive years (1999-2001) of relatively cold conditions and the development of warm anomalies in the central Pacific during the boreal summer

Corresponding author address: Ben P. Kirtman, Rosenstiel School for Marine and Atmospheric Science, University of Miami, Miami, FL 33149.

E-mail: bkirtman@rsmas.miami.edu

DOI: 10.1175/2009MWR2672.1

© 2009 American Meteorological Society

SEPTEMBER 2009 KIRTMAN AND MIN

2909

of 2002, and accurate forecasts continue to be a challenge even at relatively modest lead times (A. G. Barnston 2007, personal communication).

Despite, or perhaps because of this recent plateau of forecast skill, two notable changes in prediction strategy have emerged, largely based on multi-institutional international collaborations: (i) forecasts must include quantitative information regarding uncertainty (i.e., probabilistic prediction) and that verification must include probabilistic measures of skill (e.g., Palmer et al. 2000; Goddard et al. 2001; Kirtman 2003; Palmer et al. 2004; DeWitt 2005; Hagedorn et al. 2005; Doblas-Reyes et al. 2005; Saha et al. 2006, among many others) and (ii) a multimodel ensemble strategy may be the best current approach for adequately resolving forecast uncertainty (Palmer et al. 2004; Hagedorn et al. 2005; Doblas-Reyes et al. 2005; Palmer et al. 2008). The first change in prediction strategy naturally follows from the fact that climate variability is chaotic or irregular and, because of this, forecasts must include some quantitative assessment of this uncertainty. More importantly, the climate prediction community now understands that the potential utility of climate forecasts is based on end-user decision support (Palmer et al. 2000; Morse et al. 2005; Challinor et al. 2005), which requires probabilistic forecasts that include quantitative information regarding forecast uncertainty. The second change in prediction strategy follows from the first, because, given our current modeling capabilities, a multimodel strategy is a practical and relatively simple approach for quantifying forecast uncertainty. In fact, as argued in Krishnamurti et al. 2000, Kirtman et al. (2002b), Palmer et al. (2004), and in this manuscript, the multimodel approach appears to outperform any individual model using a standard single model approach. This may not necessarily be the case using more sophisticated single model methods such as perturbed parameters or stochastic physics.

The three objectives of this study are all related to expanding multimodel seasonal prediction capabilities. First, we document the capability of the National Center for Atmospheric Research (NCAR) Community Climate System Model, version 3 (CCSM3.0), to predict ENSO. This model is a natural candidate for inclusion in the U.S. operational multimodel prediction strategy (W. Higgins 2006, personal communication). Second, we document how CCSM3.0 can be combined with the current operational National Oceanic and Atmospheric Administration (NOAA) Climate Forecast System (CFS) to produce improved ENSO forecasts. Third, we demonstrate how an ocean initial state using one ocean-component model [i.e., the Geophysical Fluid Dynamics Laboratory (GFDL) Modular Ocean Model (MOM; Pacanowski and Griffies 1998; Derber

and Rosati 1989] can be used in a coupled system that uses a different ocean-component model [i.e., Parallel Ocean Program (POP)]. This has been done previously in terms of changing ocean model resolution (Kirtman 2003; Schneider et al. 2003), but not in terms of a complete change in ocean model formulation. This demonstration has the potential to simplify and broaden the multimodel prediction strategy by allowing institutions that do not have an independent ocean data assimilation system to more easily participate in prediction research. Unlike previous studies with the CCSM3.0, which is primarily used for climate change research, here the model is being applied to the seasonal-to-interannual prediction problem; this application is unique to this manuscript.

The deterministic verification of the SSTA forecasts presented here builds on many previous studies (i.e., Cane et al. 1986; Leetmaa and Ji 1989; Chen et al. 1995; Balmaseda et al. 1995; Ji and Leetmaa 1997; Rosati et al. 1997; Kirtman et al. 1997; Kirtman and Zebiak 1997; Stockdale 1997; Stockdale et al. 1998; Ji et al. 1994, 1996, 1998; Moore and Kleeman 1998; Schneider et al. 1999, 2003; Pope et al. 2000; Wang et al. 2002; Kirtman 2003; Alves et al. 2004; Chen et al. 2004; DeWitt 2005; Gualdi et al. 2005; Luo et al. 2005; Keenlyside et al. 2005; Saha et al. 2006; Stan and Kirtman 2008; Jin et al. 2008; Wu et al. 2009) with a particular focus on Ni�o-3.4 correlation and root-mean-square error. This is the necessary first step in developing a multimodel predictive capability. Here we highlight how the multimodel approach impacts the deterministic skill measures. Specifically, we are interested in determining where and when the multimodel approach is superior to either model alone. For probabilistic verification, we again rely on the approaches recommended in the published literature. For example, Kirtman (2003), DeWitt (2005), and Saha et al. (2006) used relative operating characteristics (ROC) scores and reliability diagrams applied to the Ni�o-3.4 forecasts. Palmer et al. (2008) and Kirtman and Pirani (2009) emphasized Brier skill scores (BSS) for tercile forecasts of surface temperature and rainfall in the so-called Giorgi regions (Giorgi and Francisco 2000). The approach we have chosen here for probabilistic verification is to present both ROC curves and reliability diagrams for the probabilistic verification of Ni�o-3.4.

2. Models, initialization strategy, and forecast experiments

Both models used in this study, the CCSM3.0 (Collins et al. 2006a,b; Kiehl et al. 2006; Large and Danabasoglu 2006; Danabasoglu et al. 2006; Meehl et al. 2006) and the CFS (Saha et al. 2006; Wang et al. 2005), are coupled

2910

MONTHLY WEATHER REVIEW VOLUME 137

ocean-atmosphere-land models whose formulations of dynamics and subgrid-scale physical parameterizations, in both the atmospheric and oceanic component models, are considered state of the art for this generation of models.

In this paper we describe results from the T85 version of CCSM3.0, with grid points in the atmospheric model [Community Atmosphere Model, version 3 (CAM3)] roughly every 1.4° latitude and longitude, and 26 levels in the vertical. The CFS data (i.e., retrospective forecasts) have been made available by NOAA (more information available online at http://cfs.ncep.noaa.gov/).

a. Ocean initialization

The ocean initialization for CCSM3.0 uses data assimilation products made available by the GFDL (Derber and Rosati 1989; A. Rosati and M. Harrison 2002, personal communication). The GFDL ocean data assimilation system is based on the MOM3 (Pacanowski and Griffies 1998) using a variational optimal interpolation scheme (Derber and Rosati 1989). MOM3 is also the ocean-component model in the CFS although the ocean data assimilation system is somewhat different from the GFDL system (see Saha et al. 2006). The GFDL ocean initial states are interpolated to the grid used in the ocean component of CCSM3.0 (i.e., POP; Collins et al. 2006b).

The following is the procedure to produce the POP restart file converted from the MOM3 ocean data assimilation restart file. The fields of the MOM3 restart file have values at time levels τ and τ + 1, while POP has data at time levels τ - 1, τ, and τ + 1. The τ - 1 time level is simply taken from the time level τ data. Both restart files have different resolutions in the horizontal and vertical. The MOM3 meridional domain covers 75°S-65°N, while the POP domain is global. The MOM3 fields have been interpolated horizontally and vertically using a bilinear interpolation scheme, which has also been used previously to reduce the resolution in MOM3-based prediction experiments (Kirtman 2003). Climatological data from long simulations of CCSM3 are used in regions where MOM3 data is undefined (i.e., poleward of 65°N and 75°S). The surface pressure for POP is estimated using the sea surface height and the pressure gradient terms are estimated using centered differencing.

b. Atmosphere, land, and sea ice initialization

The atmospheric initial states are taken from an extended atmosphere/land-only (CAM3) simulation with observed, prescribed SST. The atmospheric ensemble members were obtained by resetting the model calendar back 1 week and integrating the model forward 1 week

with prescribed observed SST. In this way, it is possible to generate an unlimited sample of initial conditions that are synoptically independent (separated by 1 week) but have the same initial date. This procedure was also used by Kirtman (2003) for ENSO prediction and Kirtman et al. (2001) to generate a 100-member ensemble for atmospheric seasonal prediction experiments.

The land initial states were also taken from these atmosphere/land extended simulations. The sea ice initial conditions were set to the climatological monthly condition based on a long simulation of CCSM3.0. No observational information is included in the sea ice initial conditions.

c. Retrospective forecast experiments

To assess the potential predictive skill of CCSM3.0, a large sample of retrospective forecast experiments have been made and compared to available observations. The retrospective forecasts cover the period 1982-98. A 12-month hindcast is initialized each 1 January, 1 April, 1 July, and 1 November¹ during this 17-yr period. For each initial month, an ensemble of 6 hindcasts is run, yielding a total of 408 retrospective forecasts to be verified. The hindcast ensembles are generated by atmospheric perturbations only and no attempt has been made to find optimal perturbations. The ocean initial state for each ensemble member is identical. We acknowledge that with this approach we may underestimate the uncertainty in any individual forecast. This limitation in our ensemble strategy is due to the fact that the ocean data assimilation was performed externally. However, the probabilistic verification identifies whether the prediction system is overconfident and can be used to quantitatively interpret any individual forecast. From a probabilistic perspective, we also may be underestimating the skill of the hindcasts by not including the uncertainty in the ocean initial states.

Throughout the ENSO prediction literature there is some confusion regarding the appropriate definition of forecast lead time. Here, forecast lead time is defined as in the following example. A CCSM3.0 forecast, initialized on 0000 UTC 1 January 1982, is labeled as being initialized in January 1982. The first monthly mean (i.e., the average of 1-31 January 1982) of the forecast is defined as the 0-month lead forecast. Similarly, the second monthly mean (i.e., 1-28 February 1982) is defined as the first month lead forecast. The CFS hindcast lead times are defined similarly, but there are notable

¹ October initial states would have been the natural choice (i.e., initial conditions every 3 months): however, in order to initiate a quasi-real-time capability in a timely manor November starts were required.

SEPTEMBER 2009 KIRTMAN AND MIN

2911

differences. For example, the forecast where January 1982 is lead time zero (February 1982 is lead time one month) is made up of 15 ensemble members. Five ensemble members are initialized during 9-13 December

1981, five members are initialized 19-23 December, and five members are initialized 30 December-3 January

1982. The remaining lead times are defined analogously. In the CFS hindcast archive this forecast is referred to the January 1982 case. In the multimodel deterministic verification discussed below we have combined, for example, the five CFS forecasts initialized on 30 December-3 January with the six CCSM3.0 forecasts initialized on 1 January. There is a trade-off here; we have chosen a CFS subensemble so that the multimodel combination is as "clean" as possible at the expense of a significantly reduced ensemble size.

3. Extended coupled simulations

Long coupled simulations sometimes provide useful insight in understanding the development of forecast errors (e.g., Kirtman 2003). In addition, we want to provide at least a preliminary assessment of how the simple initialization of CCSM versus errors in the model's ENSO impacts the forecast. Mean or systematic errors in the forecast that are similar to errors in the simulation are expected to be due to model error and largely independent of the initialization. Therefore, in this section we briefly document some key simulation errors from both CCSM and CFS that are closely linked to forecast errors. The CFS results are based on a 52-yr simulation started with observed initial conditions (January 1985) and the CCSM results are based on a 150-yr simulation started from climatology with the first 50 yr discarded as spinup.

Figure la shows the annual mean SST estimated from observations (Reynolds and Smith 1994) and the model errors in CFS (Fig. 1b) and CCSM (Fig. 1c). There are some broad features to the errors in the tropics where the model errors are fairly similar in structure, but not in terms of amplitude. Elsewhere the errors are quite different. There are notable warm biases in the stratus cloud regions of the southeastern tropical Pacific and Atlantic Oceans. In both models, errors in the tropical western Pacific and Indian Ocean are less than a degree Kelvin. Some of the largest differences in the errors occur in the extratropics. For example, CFS has a large warm bias in the Southern Ocean, this is likely due to problems in the treatment (or lack thereof) of sea ice. In the North Pacific and North Atlantic, CCSM has a large cold bias, whereas CFS has comparatively weak warm biases. Again, the differences are likely due to the treatment of sea ice.

In terms of ENSO prediction, the errors of most concern are in the near-equatorial Pacific, which are difficult to compare in Fig. 1. Figure 2 shows the equatorial Pacific (1°S-1°N) SST from the observational estimates (black line), the CFS (red line), and CCSM (blue line) simulations. From Fig. 2 it is readily apparent that CCSM has a large cold bias except in the far western and eastern Pacific. This cold bias is difficult to detect in Fig. 1, and, as will be shown later, is a significant component of the CCSM forecast systematic error. This bias also corresponds to a comparatively strong SST gradient. The equatorial SST from the CFS simulation is generally in better agreement with the observational estimates except for the far western and eastern Pacific.

The model's ability to simulate equatorial SST inter-annual variability is also relevant in terms of understanding ENSO forecast skill. Here we show time-longitude monthly SST anomaly (SSTA) cross sections along the equator in the Pacific from the two simulations (Figs. 3a,b) and from the observational estimates (Fig. 3c). Since these plots are based on extended simulations, we cannot compare the SSTA point by point�we can only make qualitative assessments. Both models produce SST variability that bears some resemblance to the observations. There are, however, some errors worth noting in terms of understanding the forecasts shown later. For example, CFS has considerably more variance than observed, the duration of the events is longer than the observed estimates, and there is a tendency for the simulation to be too regular (H. Pan 2007, personal communication). The CCSM exhibits a tendency for warm and cold events that are too short compared to the observed, there is too much power on biennial time scales, and an erroneous westward propagation. Although it is difficult to detect in this figure, the ENSO events in CCSM tend to extend too far into the western Pacific compared to the observational data (please see Deser et al. 2006, their Figs. 3 and 5). This aspect of the variability will be easy to detect in the forecast evolution discussed later.

4. Forecast results

In terms of assessing the forecast skill we have removed the systematic error from both prediction systems in the same way. The systematic error of the hindcasts is defined as the average of all the forecasts (minus the observed monthly climatology) for that particular initial month and as a function of lead time. For example, consider the six-member CCSM forecasts initialized each January from 1982 to 1998. The systematic error in the January initial condition forecast SST is defined as

2912

MONTHLY WEATHER REVIEW VOLUME137

FIG. 1. (a) Annual mean SST (1971-2000) based on observational estimates (°C): and model annual mean SST errors (i.e., model minus observations), (b) CFS (52-yr simulation), and (c) CCSM (100-yr simulation).

where k refers to the ensemble member, yr corresponds to the initial condition year, τ indicates lead time, and SST_obs(τ) is the observed monthly climatology. This is the same approach used in Saha et al. (2006), Kirtman (2003), and Kirtman et al. (1997). To calculate the systematic error from the CFS forecasts we have applied the same approach. We have also compared the systematic error using both the full 15-member ensemble

and the 5-member subensemble discussed earlier and found that it has little impact on the results presented here. In either case, we use the same 17-yr period for calculating the systematic error and for the verification. The systematic error for the eastern Pacific SSTA will be discussed later in the paper.

To highlight some elements of the simulations that are also present in the retrospective forecasts, we show some specific examples. Where possible we attempt to identify consistent forecast errors that are due to model error as opposed to initialization error. The reason for this is to document how the differences in model errors

SEPTEMBER 2009 KIRTMAN AND MIN

2913

FIG. 2. Equatorial Pacific SST from observational estimate (°C. black line), the CFS simulation (red line), and the CCSM simulation (blue line).

impact differences in forecast evolution. We do not intend to suggest that observational or initialization error is not important. Figure 4 shows time-longitude equatorial Pacific SSTA cross sections for each CCSM3.0 ensemble member and for six randomly chosen ensemble members from the CFS hindcast dataset. The first two columns correspond to the CCSM3 forecasts with Fig. 4a showing the observational estimate and Fig. 4b showing the ensemble mean. The various CCSM3 forecast are denoted with Figs. 4c-h. Similarly, the second two columns correspond to the CFS forecast with Fig. 4a showing the observational estimate and Fig. 4b showing the ensemble mean. The various CCSM3 forecasts are denoted with Figs. 4c-h. In this example, January corresponds to lead time zero and the CFS forecasts are only run through September (lead time of 8 months). Both sets of forecasts capture the development of the warm event, although they also have a tendency to initiate the warming a few months too early. The CCSM forecasts are somewhat more confident in the development of the warm event, which may not be a positive attribute particularly in terms of probabilistic verification. The excessive westward propagation and extension into the far western Pacific is also readily apparent in the CCSM forecast as well as in the extended simulation, and therefore is likely due to model error as opposed to initialization error. The CFS forecasts appear to capture some indication of the rapid intensification of the event during the late boreal summer and early fall.

Figure 5 is in the same format as Fig. 4 except for the January 1983 case. As in Fig. 4, the excessive westward propagation and extension in the CCSM forecasts is apparent. The CCSM forecasts quite consistently cap-

ture the transition from warm to cold SSTA. This may be fortuitous and due to the fact that CCSM has too strong a biennial tendency. On the other hand, there are indications that CFS has difficulty capturing the transition from warm to cold, which may be related to the fact that, in the CFS extended simulation, the duration of the ENSO events is too long.

An example from the 1988 cold event is shown in Fig. 6. In the ensemble means, both models predict the development of the cold event. Again, we see considerably more confidence in the development of strong cold anomalies in CCSM. Both models also show a tendency for the cold anomalies to develop too early relative to the observed estimates. This is consistent with the warm event shown in Fig. 4, and is common among many forecast systems (e.g., Kirtman et al. 2002a). Essentially, given any western Pacific subsurface preconditioning (i.e., heat content anomaly), many models tend to immediately forecast the eastward migration of that anomaly and the rapid subsequent development of the SSTA in the eastern Pacific (e.g., Kirtman et al. 1997; Schneider et al. 1999; Kirtman 2003; Vintzileos et al. 2005).

a. Deterministic verification

As mentioned in the introduction, the goal of the paper is to evaluate CCSM and CFS together as a potential multimodel prediction system. The deterministic verification presented here emphasizes the SSTA correlation of the forecasts with the observations and the root-mean-square error (RMSE). These results are presented for Nino-3.4 (area averaged SSTA from 5°S-5°N to 170°-120°W) as a function of lead time and as a point-by-point calculation for each ocean grid point on the globe.

Part of the verification process involves removing the systematic error in the forecast as discussed above. Here we diagnose the systematic error by plotting the Ni�o-3.4 systematic evolution for all four initial condition months (Figs. 7a-d). Generally, the mean evolution of the CFS forecasts is in better agreement with the observations than the CCSM forecasts. The CCSM forecasts suffer from a strong cold bias that is apparent regardless of lead time, including lead time zero. One interesting aspect of the CFS forecasts is that the systematic errors become larger during September and are quite cold. The CCSM forecasts systematically produce Ni�o-3.4 temperatures that are too cold. This is quite systematic and is independent of sampling issues.

Figure 8 shows the Nino-3.4 time series from all CCSM3.0 and CFS forecasts with January initial conditions. Overall, both forecast systems capture the major warm and cold events, but there are important

2914

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 3. Time-longitude cross section of the SSTA (°C) along the equator in the equatorial Pacific, (a) SSTA from the CFS simulation.

(b) the CCSM simulation, and (c) the observed estimates.

shortcomings that should be noted. Some of the CFS cases produce erroneous second warmings at relatively long leads, as in the 1983, 1987, and 1990 cases. The marked contrast between the CFS and CCSM forecast initialized in January 1987 is also noteworthy. Establishing how systematic these errors in forecast evolution are would require several more years of forecast cases.

Figures 9a-e show the correlation coefficients for the six-member CCSM ensemble mean forecasts (blue curve), the ensemble mean five-member CFS subensemble (red curve), and the multimodel ensemble mean (black curve).

The multimodel ensemble mean is the average of all 11 members of the ensemble with no weighting applied. The correlation coefficients are calculated separately for each initial condition month (i.e., Figs. 9a-d) and the average correlation is shown in Fig. 9e. There are several points to note:

(i) The correlation coefficients are quite comparable for all lead times and initial months and are almost indistinguishable for the July cases. Arguably. CCSM performs more skillfully for the January

SEPTEMBER 2009 KIRTMAN AND MIN

2915

FIG. 4. Time-longitude equatorial Pacific SSTA cross sections for each CCSM3.0 ensemble member and for six randomly chosen ensemble members from the CFS hindcast dataset. The first two columns correspond to the CCSM3 forecasts with (a) the observational estimate and (b) the ensemble mean, (c)-(h) Various CCSM3 forecast are denoted. Similarly, the last two columns [also labeled (a)-(h)] correspond to the CFS forecast with (a) the observational estimate and (b) the ensemble mean, (c)-(h) The various CCSM3 forecasts are denoted. In this example, the lead time zero forecast is for January 1982.

2916

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 5. As in Fig. 4. but the lead time zero forecast is for January 1983.

SEPTEMBER 2009 KIRTMAN AND MIN

2917

FIG. 6. As in Fig. 4, but the lead time zero forecast is for January 1988.

2918

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 7. Ni�o-3.4 forecast systematic evolution (i.e., average of all forecasts initialized in a particular month) as a function of lead time. The CCSM forecasts are noted in blue and the CFS forecasts are in red. The observed mean evolution of Ni�o-3.4 temperatures is shown in black. The forecasts are initialized in (a) January, (b) April, (c) July, and (d) November.

cases and CFS performs more skillfully for the April cases. In terms of a multimodel average, the fact that the models have similar skill levels is a positive attribute. The most desirable feature would be similar but complimentary skill, (ii) The multimodel ensemble mean (black curve) has the highest correlation for most lead times independent of initial month.

(iii) Most notably the large drop in skill for January CFS forecasts for lead times 4-6 (i.e., spring prediction barrier: see also Wu et al. 2009) has only a small impact on the multimodel skill.

(iv) The correlation coefficient is largest for the July forecasts as expected from previous studies (e.g., Kirtman 2003).

The RMSE is also a useful metric of forecast skill particularly when considered in conjunction with the correlation coefficient. In the same format as Figs. 9a-e, Figs. 10a-e shows the RMSE as a function of lead time for all the forecasts. Much like the correlation, the RMSE is generally improved with the multimodel ensemble (i.e., the unweighted mean of the forecasts from both models), and qualitatively speaking, the errors for both models are quite comparable. Arguably, the CCSM

performs more skillfully for January and November initial conditions and the CFS performs more skillfully for the April and July initial conditions. Part of the multi-model improvement is merely due to the larger ensemble size (11 members versus 5 or 6), but a significant component of the reduction is due to the multimodel aspect. This is discussed in more detail later.

For some lead times and some initial months CFS has higher skill scores than CCSM or than the multimodel ensemble. Similarly, sometimes the CCSM forecasts have higher hindcast skill. The multimodel approach "smoothes out" these relatively large variations in skill much more efficiently than any one single model. This is supported by the correlation (Fig. 9e) and the RMSE (Fig. 10e) calculated for all initial months for the results shown in Figs. 9a-d and 10a-d. The all-initial-months correlation from the multimodel ensemble is as high as or higher then the best model for all lead times. We acknowledge that the separation is relatively small; however, when considering the RMSE we see a more dramatic reduction in the error. Later in the manuscript (see discussion of Figs. 15 and 16) we address how the multimodel approach impacts the correlation and RMSE as opposed to simply increasing ensemble size of only one model.

SEPTEMBER 2009 KIRTMAN AND MIN

2919

FIG. 8. Ni�o-3.4 time series for the CCSM3.0 ensemble forecasts (blue) and the CFS ensemble forecasts (red).

It is arguably the standard practice to issue Ni�o-3.4 SSTA bias corrected forecasts (e.g., the Ni�o-3.4 forecast plume posted by the International Research Institute; see online at http://iri.columbia.edu/climate/ENSO/ currentinfo/SST_table.html). However, the atmospheric teleconnections necessary depend on the uncorrected SSTA. As such, we show (Fig. 11) the Nino-3.4 correlation and RMSE for both models and the multimodel ensemble where the anomalies are defined with respect to the observed climatology. The correlation and RMSE are shown for the forecasts initialized in January. As expected, the correlation for the individual models falls off considerably faster then with the bias corrections (see Fig. 9a). The multimodel correlation is at times larger then either model and at times lies between the two. The RMSE for the CCSM forecasts grow quite rapidly compared to either the CFS forecasts or the multimodel forecasts. Given the systematic errors shown in Fig. 7a this is to be expected. The multimodel RMSE, however, is quite close to the CFS RMSE for the first 7 months of the forecast period.

While the focus of this paper is on Ni�o-3.4 skill scores, Fig. 12 shows that the global distribution of the point correlation for the predicted SST and the observed estimates for both models and the multimodel ensemble. The forecasts are initialized in January and

are verified for the following June (i.e., five-month lead). For this particular lead time and initial condition, the CCSM forecasts appear to be more skillful in the tropical Pacific, and much of this skill is captured in the multimodel ensemble. Conversely, the CFS has higher skill in the Atlantic and the multimodel ensemble appears to also capitalize on this skill, and hints at even higher skill. The correlation is comparable between CCSM and CFS in the Indian Ocean and is maintained in the multimodel ensemble.

Figure 13 is in the same format as Fig. 12, but for forecasts initialized in July. As expected, the correlation coefficients in the tropical Pacific are considerably higher than for the January forecasts, whereas the skill in the Atlantic is reduced relative to the January forecasts. The correlation in the Indian Ocean is similar in the January and July forecasts. Any differences in the correlation among CCSM, CFS, and the multimodel are not statistically significant.

b. Assessing the statistical significance of the multimodel ensemble

From a statistical perspective, it is possible that the multimodel combination generally improves the forecast skill via random chance. To examine this question, we performed a series of Monte Carlo calculations

2920

MONTHLY WEATHER REVIEW VOLUME137

FIG. 9. Ni�o-3.4 correlation coefficient for ensemble mean forecasts initialized in (a) January, (b) April, (c) July, and (d) November, (e) The correlation coefficient calculated over all the cases. The red curves correspond to CFS, the blue curves correspond to CCSM. and the black is the multimodel ensemble.

forming a multimodel combination with one dynamic model (CFS or CCSM) and a red noise model. The red noise model is an auloregressive model [AR(l)] based on the particular CGCM-simulated Ni�o-3.4 variability. Here we show the Ni�o-3.4 correlation coefficient for forecasts initialized in January for a lead time of 5 months (Fig. 14). The January forecasts were chosen specifically because the models have so much difficulty predicting through the boreal spring season, and thus it is a robust challenge for the multimodel formulation.

The correlation for the 6-member ensemble of CCSM is shown in blue, the 5-member CFS ensemble is in red, and the 11-member multimodel ensemble is shown in green. The black and gray bars assess the significance of the multimodel ensemble. For example, the black bars indicate the correlation coefficient of the "multimodel" ensemble where one model is the CFS forecast and the other is random red noise based on fitting the AR(1) model. The correlation coefficient for this CFS + noise multimodel ensemble is plotted for 1000 different sets of

SEPTEMBER 2009 KIRTMAN AND MIN

2921

FIG. 10. As in Fig. 9. but for RMSE.

noise realizations. The black bars exceed the CFS + CCSM multimodel ensemble just once in the 1000 trials, indicating that by adding CCSM to CFS there is 99.9% confidence that the correlation exceeds random chance.

The case where CCSM is the baseline model is more interesting. For this particular lead time and initial month, the multimodel ensemble skill is actually lower than the CCSM skill alone. In this case, the Monte Carlo simulation addresses the question of whether adding CFS to CCSM degrades the skill as much as simply adding noise. The gray bars in Fig. 14 indicate the correlation for a CCSM + noise multimodel en-

semble. In this case there is 97% confidence that adding CFS to CCSM degrades the correlation less than would be expected from merely adding noise (e.g., 30 of the gray bars exceed the multimodel skill). It needs to be emphasized that there are examples where the relative roles of CCSM and CFS are reversed, and more importantly, for most initial months and lead times, the multimodel skill is higher than either model alone (as shown in Figs. 9e and 10e). In fact, in most cases the multimodel ensemble skill is higher than either model alone with 95% confidence according to this simple testing procedure.

2922

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 11. Ni�o-3.4 (top) correlation coefficient and (bottom) RMSE for ensemble mean forecasts initialized in January. In this case, the anomalies are taken with respect to the observed climatology (i.e.. no bias correction). The red curves correspond to CFS. the blue curves correspond to CCSM, and the black is the multi-model ensemble.

c. Is a multimodel ensemble superior to larger single model ensembles?

Figure 15 provides an indication that the multimodel ensemble is superior to simply increasing the ensemble size of the CFS forecasts. Here we have plotted Ni�o-3.4 correlation coefficients for forecasts initialized in January at a lead time of 5 months. The 5-member CFS correlation is shown in red, the 6-member CCSM correlation is shown in blue, and the multimodel (11 member) correlation is shown in green. The black curve shows all

possible five-member ensembles drawn from the six-member CCSM forecasts and the five-member CFS forecasts with the condition that at least two members must come from each model. All possible combinations of five-member multimodel ensembles is well separated from the five-member CFS correlation suggesting that increasing the CFS ensemble size will not be competitive with the multimodel ensemble (Hagedorn et al. 2005).

Figure 16 shows (in a format similar to Fig. 15) the RMSE for all possible five-member multimodel ensembles drawn from both CCSM and CFS using the same rules discussed above. Recall that the question we are trying to address is whether the multimodel ensemble improves the skill simply because the ensemble size is larger. To assess this, first consider the case where the CFS is used as the benchmark, then clearly a multi-model ensemble consisting of five members is superior to a five-member CFS ensemble even if only two ensemble members are drawn from CCSM for this particular initial month and lead time. This conclusion is based on the fact that the black curve is well separated from the red line. If CCSM is used as the benchmark, then obviously the multimodel ensemble of comparable ensemble size has a larger RMSE for this particular lead time and initial month.

Again, we remind the reader that it is our assertion that the multimodel is in general an improvement of the individual model. As expected, there are going to be exceptions. As a final example of the impact of the multimodel ensemble, we have also simply compared all possible multimodel 11-member ensemble combinations of CCSM + CFS against all possible 11-member ensembles from CFS alone. Over 93% of the time the multimodel ensemble has a higher Ni�o-3.4 correlation coefficient.

d. Traditional significance testing

The forecasting approach advocated here is, obviously, a multi-CGCM-based approach. Nevertheless, a comparison with standard statistical forecasting techniques provides useful benchmarks for evaluating the CGCM forecasts (e.g., van Oldenborgh et al. 2005). Here we briefly make comparisons of the correlation skill scores among the CGCM forecasts (i.e., CFS, CCSM, and the multimodel) with a readily available operational statistical model (Markov; Xue et al. 2000). We also provide 95% confidence intervals calculated using bootstrap method with moving block lengths equal to the forecast error decorrelation time. This is the same approach van Oldenborgh et al. (2005) applied to ECMWF forecasts. These confidence intervals are another metric that can be used to evaluate the statistical

SEPTEMBER 2009 KIRTMAN AND MIN

2923

FIG. 12. SSTA local point correlation for the (a) CCSM ensemble (6 members), (b) the CFS ensemble (5 members), and (c) the multimodel ensemble (11 members). The plots correspond to a lead time of 5 months and January initial conditions.

significance (or lack there of) of the differences between the individual models and the multimodel mean. (The multimodel mean is restricted to just CCSM and CFS). The results are summarized in Table 1 for lead times of 3, 5, and 8 months.

At a lead time of 3 months the difference in correlation for the multimodel forecast and the CCSM forecast is significant at 95%, but the difference is not significant at 95% with CFS (although it is significant at 90%-not shown). In contrast, the multimodel correlation is significantly different at 95% from the CFS forecasts at the 5-month lead, whereas the correlation for CCSM is within the confidence interval. For a lead time of 8 months the correlations are indistinguishable at the 95% level.

These confidence intervals are consistent with our interpretation that the multimodel forecast is generally indistinguishable from the best model while which model is best model appears to be a function of lead time and initial month. Finally, for lead times of 3 and 5 months the multimodel forecast is significantly better than the Markov model forecasts at the 95% level and they are indistinguishable at the 8-month lead.

e. Ni�o-3.4 probabilistic verification

As mentioned in the introduction, ensemble prediction systems are routinely used to estimate forecast uncertainty or to provide probability distributions. One of the arguments in favor of the multimodel approach is

2924

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 13. As in Fig. 12, but for a lead time of 5 months and July initial conditions.

that multiple models may do a better job of resolving the probability because they probe the uncertainty due to the physical parameterizations, albeit in an ad hoc fashion. Here we examine this question by verifying the forecasts using probabilistic measures of skill.

There are a number of possible ways to verify probabilistic forecasts. Here we have chosen methods suggested by the World Meteorological Organization (WMO) standardized verification system (SVS) for long-range forecasts (LRF; see online at http://www.meloffice. gov.uk/research/seasonal/SVSLRF.pdf), and we focus on ROCs (see also Mason and Graham 1999; Kirtman 2003; DeWitt 2005; Stan and Kirtman 2008) and reliability (see also Saha et al. 2006).

The ROC calculation is based on hit rates and false alarm rates that are calculated by generalizing a standard 2x2 contingency table to probabilistic ensemble forecasts. For example, a "warning" for a probabilistic forecast is issued when the number of ensemble members forecasting the event exceeds some threshold. The 2x2 contingency table can then be calculated for each threshold and the hit rates and false alarm rates can also be calculated. The ROC curve is then formulated by plotting the hit rate versus the false alarm rate for each threshold. The ROC curves for the Ni�o-3.4 hindcasts for the upper (warm events) and lower (cold events) terciles for a lead time of 6 months is shown in Fig. 17. In calculating the ROC curves, the corresponding

SEPTEMBER 2009 KIRTMAN AND MIN

2925

FIG. 14. Ni�o-3.4 correlation coefficient at a lead time of 5 months for the 6-member CCSM ensemble (blue), the 5-member CFS ensemble (red), the multimodel ensemble (green), the CCSM ensemble plus noise (gray), and the CFS ensemble plus noise (black). The oval highlights the fact that only one black bar in the entire 1000 samples has a correlation coefficient exceeding the value of the multimodel ensemble.

contingency tables have been aggregated over all model grid points in the Ni�o-3.4 region. The retrospective forecasts and the observations have been normalized by their respective standard deviation.

An ideal probabilistic forecast system would have relatively large hit rates and small false alarm rates so that all the points on the ROC curve would cluster in the upper-left corner of the diagram. For a relatively poor

FIG. 15. Ni�o-3.4 correlation coefficient at a lead time of 5 months. The black curve shows the correlation for all possible multimodel combinations where at least two ensemble members are drawn from CCSM and two ensemble members are drawn from CFS. The red curve corresponds to the skill of the complete five-member CFS ensemble and the blue curve corresponds to a randomly chosen five-member ensemble from CCSM.

2926

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 16. Ni�o-3.4 RMSE at a lead time of 5 months. The black curve shows the RMSE for all possible multimodel combinations where at least two ensemble members are drawn from CCSM and two ensemble members are drawn from CFS. The red curve corresponds to the skill of the complete five-member CFS ensemble and the blue curve corresponds to a randomly chosen five-member ensemble from CCSM.

forecast system, all the points on the ROC would lie very close to the diagonal line indicating that the hit rates and false alarm rates are nearly the same (i.e., no skill). The area under the ROC curve above the diagonal line is often used as a single index of probabilistic skill.

The interior points on the ROC curve indicate the number of ensemble members forecasting the particular event. Progressing from the lower-left corner to the upper-right corner, the first interior point on, say the black curve, corresponds to 11 out of 11 members of the multimodel ensemble forecasting the event. This point indicates how skillful the multimodel ensemble system is when all the members agree. Not surprisingly, the hit rates are low, but the false alarm rates are even lower, suggesting significant skill. The second point corresponds to 10 out of 11 members forecasting the event, and the last interior point corresponds to 1 out of 11 members forecasting the event. In this case, the hit rates and false alarm rates are relatively high, but if the hit rate is higher than the false alarm rate then the system has some skill. Similar arguments apply for the six-member ensemble CCSM ROC curve shown in blue and the five-member ensemble CFS ROC curve shown in red.

For warm events (top panel of Fig. 17), the multi-model ROC curve is separated from both the CFS and CCSM curves, and is closer to the upper-left corner indicating greater ROC area. For highly confident forecasts (i.e., almost all the ensemble members forecasting the warm event), CFS and CCSM have comparable hit

rates and false alarm rates, which are significantly separated from the no-skill diagonal. When the number of ensemble members forecasting the event decreases there is some indication that the CCSM has higher hit rates, but with similar false alarm rates. For cold events (bottom panel of Fig. 17). the multimodel appears to be more skillful (i.e.. higher hit rates) than either CFS or CCSM when typically fewer than half the ensemble members forecast the event. There is also an indication that the CFS has larger ROC area then CCSM for cold events. Overall, the ROC curves are consistent with the deterministic measures of skill: CFS and CCSM are comparable and the multimodel is generally more skillful than either model alone.

Following the WMO SVS recommendations, we have also calculated reliability diagrams (Fig. 18). Reliability diagrams are particularly sensitive to small sample sizes such as is the case here, so some caution must be exercised in interpreting the results. To formulate the reliability diagram, the expectation of the observed probability given the forecast probability [i.e., observed relative frequency; denoted E(P₀|P_f)] is plotted against the forecasted probability (P_f). A perfectly reliable forecast system would have its probability forecasts corresponding with the observed relative frequency and, therefore would display a 1:1 diagonal line for each event. An overconfident forecast system would tend to have forecast probabilities that are higher than the observed relative frequency.

SEPTEMBER 2009 KIRTMAN AND MIN

2927

In terms of warm events (top panel of Fig. 18), the CFS is generally overconfident (consistent with the reliability shown in Saha et al. 2006), whereas the multi-model tends to reduce this overconfidence except in the cases of large forecast probabilities. For small forecasts probabilities of warm events, both CFS and CCSM underforecast, and again, the multimodel tends to improve the reliability in this regard. Similar improvements for small forecast probabilities of cold events (bottom panel of Fig. 18) are also noted. It is unclear whether the overall multimodel reliability is higher for cold events. These probabilistic measures of forecast skill are quite comparable to the published literature (e.g., Saha et al. 2006: Kirtman 2003, among others).

5. Summary and conclusions

The purpose of this paper was to (i) describe a set of seasonal-to-interannual predictions made with the NCAR CCSM3.0, (ii) assess the performance of the multimodel ensemble formed using CFS and CCSM together, and (iii) demonstrate how an ocean initial state from one component model can be used in a coupled system that uses a different ocean component. These two particular models were chosen for pragmatic reasons�they are U.S. national models; one is already operational and the other is extensively used for climate change research. Nevertheless, there is some additional qualitative a posteriori justification for this choice; namely, that the models have complementary systematic biases (i.e., one is relatively warm along the equator and one is relatively cold) and complementary differences in their ENSO statistics (i.e., one is biennial the other quadrennial and the events are too long in one and too short in the other). It is our hope that these complementary biases lead to complementary forecast skill, which acts to improve the skill in a multi-model prediction system. In fact, only complementary skill is genuinely useful and additional work will be done to quantitatively identify complementary forecast skill.

The initialization of the CCSM coupled system follows a very simple procedure of interpolating existing ocean analyzes to the coupled model grid. This has the advan-

tage of allowing many more models to participate in the prediction problem. However, we speculate that additional improvements in forecast skill will follow from careful examination of the coupled initialization problem. Can we use the ocean analysis and the coupled model to find the ocean initial state that produces the best forecast for a given model or multimodel ensemble? We also expect improvements over land by initializing the land state in the coupled model and some effort in this regard with CCSM is ongoing.

Overall, both models are quite comparable in forecast skill. There are lead times and initial months were we

FIG. 17. ROC for Ni�o-3.4 predictions. The ROC curve for (top) warm events (i.e.. the upper tercile) and (bottom) cold events (i.e.. bottom tercile). The multimodel ensemble is shown in black. CFS in red. and CCSM in blue.

2928

MONTHLY WEATHER REVIEW VOLUME 137

FIG. 18. Reliability diagrams for Nino-3.4 predictions. The reliability for (top) warm events (i.e.. the upper tercile) and (bottom) cold events (i.e.. bottom tercile). The multimodel ensemble is shown in black. CFS in red. and CCSM in blue. The x axis corresponds to the forecast probability and the y axis corresponds to the observed relative frequency.

can show that CFS is statistically significantly better than CCSM and there are lead times and initial months were we can show that CCSM is statistically significantly better then CFS. Therefore, we make no conclusions regarding which model is better, we are simply interested in whether the combination of the two improves the forecast skill. There are also lead times and initial months where the best model (either CCSM or CFS) is superior to the multimodel ensemble, and there are lead times and initial months where the skill is statistically indistinguishable among the three. We calculated the

classical 95% confidence intervals for the Ni�o-3.4 correlation coefficients at lead times of 3.5, and 8 months. Here we find that multimodel forecast is statistically indistinguishable from the best model while which model is best model appears to be a function of lead time and initial month. Moreover, the multimodel forecast does appear to produce correlations that are significantly higher than the worst model, which again is a function of lead time and initial month. It is this fact that leads us to the conclusion that the multimodel improves the "overall" forecast skill and emphasizes how the multimodel ensemble can conceptually be thought of as smoothing out the vagaries in skill associated with individual model differences.

Determining where and when the multimodel approach is superior to either model alone is difficult to answer without serious caveats. Suppose we define "superior" to be statistically indistinguishable from the best model, but significantly better than the worst model. Then we can conclude that for relatively short lead times (i.e., 0-5 months) the multimodel approach is superior when considering all initial months. At longer leads the differences in the skill are statistically indistinguishable. While it was not shown here, we have calculated the statistical significance as a function of initial month, and, for the forecasts initialized in July, the multimodel forecast skill is indistinguishable from the individual models at all lead times. The remaining initial months have the same significance relationships as the collection of all months.

We also examined the Ni�o-3.4 forecast skill from a probabilistic perspective (i.e., reliability and ROC curves). The general impression was that the multimodel increased the ROC area and smoothed the reliability for warm events. For cold events, the multimodel ensemble had smaller impact probabilistic skill, but was generally positive.

It is important to note that the multimodel approach is not magical. As expected, there are lead times and initial months when one particular model is superior to the other model and the multimodel ensemble. The multimodel approach increases the ensemble size by probing model uncertainty as well as initial condition uncertainty, which tends to smooth out the forecast skill and reduce the impact of systematic "misfires" associated with any one particular model. Continuing research in multimodel calibration activities is yielding positive results (L. Goddard 2007, personal communication).

It is also important to note that the multimodel approach, at least as implemented here, is largely ad hoc. Much additional work needs to be done to identify the optimal approach to fully resolving the probability distribution and combining the models. Finally, we urge the climate modeling community to recognize that the

SEPTEMBER 2009 KIRTMAN AND MIN

2929

multimodel approach does not obviate the need to improve models. Many of the model errors noted here continue to reduce the forecast skill and the ability to use the forecasts for societal benefit. In fact, the reader is reminded that the SSTA forecasts evaluated here have had their respective systematic errors removed. The systematic error has large impacts on the location of tropical convection and the associated teleconnections, which impacts whether the forecasts are of any societal benefit. This issue must be addressed in the future.

Acknowledgements. This research was supported by the Office of Science (BER), U.S. DOE DE-FG02-07ER64473, NSF Grants ATM-0653123, OCI-0749165, and ATM-0754341 and NOAA Grants NA17RJ1226 and NA080AR4320889. The authors are grateful to the University of Miami Center for Computational Science for resources needed to complete the project.

REFERENCES

Alves, O., M. A. Balmaseda, D. Anderson, and T. Stockdale, 2004: Sensitivity of dynamical seasonal forecasts to ocean initial conditions. Quart. J. Roy. Meteor. Soc., 130, 647-667.

Balmaseda, Μ. Α., Μ. Κ. Davey, and D. L. T. Anderson, 1995: Decadal and seasonal dependence of ENSO prediction skill. J. Climate, 8, 2705-2715.

Barnston, A. G., M. Glantz, and Y. He, 1999: Predictive skill of statistical and dynamical climate models in SST forecasts during the 1997-98 El Ni�o and the 1998 La Ni�a onset. Bull. Amer. Meteor. Soc., 80, 217-243.

Cane, Μ. Α., S. Ε. Zebiak, and S. C. Dolan, 1986: Experimental forecasts of El Ni�o. Nature, 321, 827-832.

Challinor, A. J., J. M. Slingo, T. R. Wheeler, and F. J. Dobias Reyes, 2005: Probabilistic simulations of crop yield over western India using the DEMETER seasonal hindcast ensembles. Tellus, 57A, 498-512.

Chen, D., S. E. Zebiak, A. J. Busalacchi, and M. A. Cane, 1995: An improved procedure for El Ni�o forecasting: Implications for predictability. Science, 269, 1699-1702.

�, M. A. Cane, A. Kaplan, S. E. Zebiak, and D. Huan, 2004:

Predictability of El Ni�o over the past 148 years. Nature, 428, 733-736.

Collins, W. D., and Coauthors, 2006a: The Community Climate System Model version 3 (CCSM3). J. Climate, 19, 2122-2143.

�, and Coauthors, 2006b: The formulation and atmospheric

simulation of the Community Atmosphere Model Version 3

(CAM3). J. Climate, 19, 2144-2161. Danabasoglu, G., W. G. Large, J. J. Tribbia, P. R. Gent,

B. P. Briegleb, and J. C. McWilliams, 2006: Diurnal coupling

in the tropical oceans of CCSM3. J. Climate. 19, 2347-2365. Derber, J., and A. Rosati, 1989: A global oceanic data assimilation

system. J. Phys. Oceanogr., 19, 1333-1347. Deser, C., A. Capotondi, R. Saravanan, and A. S. Phillips, 2006:

Tropical Pacific and Atlantic climate variability in CCSM3.

J. Climate, 19, 2451-2481. DeWitt, D. G., 2005: Retrospective forecasts of interannual sea

surface temperature anomalies from 1982 to the present using

a directly coupled atmosphere-ocean general circulation model. Mon. Wea. Rev., 133, 2972-2995.

Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting�II. Calibration and combination. Tellus, 57A, 234-252, doi:10.1111/j.l600-0870.2005.00104.x.

Giorgi, F., and R. Francisco, 2000: Uncertainties in regional climate change prediction: A regional analysis of ensemble simulations with the HADCM2 coupled AOGCM. Climate Dyn., 16, 169-182.

Goddard, L., S. J. Mason, S. E. Zebiak, C. F. Ropelewski, R. Basher, and M. A. Cane, 2001: Current approaches to seasonal-to-interannual climate predictions. Int. J. Climatol., 21, 1111-1152.

Gualdi, S., A. Alessandri, and A. Navarra, 2005: Impact of atmospheric horizontal resolution on El Ni�o-Southern Oscillation forecasts. Tellus, 57A, 357-374.

Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting�I. Basic concept. Tellus, 57A, 219-233, doi:10.1111/j.1600-0870.2005.00103.x.

Ji, M., and A. Leetmaa, 1997: Impact of data assimilation on ocean initialization and El Ni�o prediction. Mon. Wea. Rev., 125, 742-753.

�, A. Kumar, and A. Leetmaa, 1994: An experimental coupled

forecast system at the National Meteorological Center: Some early results. Tellus, 46A, 398-418.

�, A. Leetmaa, and V. E. Kousky, 1996: Coupled model forecasts of ENSO during the 1980s and 1990s at the National Meteorological Center. J. Climate, 9, 3105-3120.

�, D. W. Behringer, and A. Leetmaa, 1998: An improved

coupled model for ENSO prediction and implications for ocean initialization. Part II: The coupled model. Mon. Wea. Rev., 126, 1022-1034.

Jin, Ε. K., and Coauthors, 2008: Current status of ENSO prediction skill in coupled ocean-atmosphere models. Climate Dyn., 31, 647-664, doi:10.1007/S00382-008-0397-3.

Keenlyside, N., M. Latif, M. Botzet, J. Jungclaus, and U. Schulzweida, 2005: A coupled method for initializing El Ni�o-Southern Oscillation forecasts using sea surface temperature. Tellus, 57A, 340-356.

Kiehl, J. T., C. A. Shields, J. J. Hack, and W. D. Collins, 2006: The climate sensitivity of the Community Climate System Model Version 3 (CCSM3). J. Climate, 19, 2584-2596.

Kirtman, B. P., 2003: The COLA anomaly coupled model: Ensemble ENSO prediction. Mon. Wea. Rev., 131, 2324-2341.

�, and S. E. Zebiak, 1997: ENSO simulation and prediction

with a hybrid coupled model. Mon. Wea. Rev., 125, 2620-2641.

�, and A. Pirani, 2009: The state of the art of seasonal prediction. Bull. Amer. Meteor. Soc., 90, 455-458.

�, J. Shukla, B. Huang, Z. Zhu, and Ε. K. Schneider, 1997:

Multiseasonal prediction with a coupled tropical ocean global atmosphere system. Mon. Wea. Rev., 125, 789-808.

�, D. A. Paolino, J. L. Kinter III, and D. M. Straus, 2001: Impact of tropical subseasonal SST variability on seasonal mean climate simulations. Mon. Wea. Rev., 129, 853-868.

�, Y. Fan, and Ε. K. Schneider, 2002a: The COLA global

coupled and anomaly coupled ocean-atmosphere GCM. J. Climate, 15, 2301-2320.

�, J. Shukla, M. Balmaseda, N. Graham, C. Penland, Y. Xue,

and S. Zebiak, 2002b: Current status of ENSO forecast skill: A report to the Climate Variability and Predictability (CLIVAR) Numerical Experimentation Group (NEG).

2930

MONTHLY WEATHER REVIEW VOLUME 137

CLIVAR Working Group on Seasonal to Interannual Prediction, 31 pp.

Krishnamurti, T. N., C. M. Kishtawal, Z. Zhang, T. E. LaRow, D. A. Bachiochi, C. E. Williford, S. Gadgil, and S. Surendran, 2000: Multimodel ensemble forecasts for weather and seasonal climate. J. Climate, 13, 4196-4216.

Landsea, C. W., and J. A. Knaff, 2000: How much skill was there in forecasting the very strong 1997-98 El Ni�o? Bull. Amer. Meteor. Soc., 81, 2107-2120.

Large, W. G., and G. Danabasoglu, 2006: Attribution and impacts of upper-ocean biases in CCSM3. J. Climate, 19, 2325-2346.

Leetmaa, Α., and M. Ji, 1989: Operational hindcasting of the tropical Pacific. Dyn. Atmos. Oceans, 13, 465-490.

Luo, J. J., S. Masson. S. Behera, S. Shingu, and T. Yamagata, 2005: Seasonal climate predictability in a coupled OAGCM using a different approach for ensemble forecasts. J. Climate, 18, 4474-497.

Mason, S. J., and Ν. E. Graham, 1999: Conditional probabilities, relative operating characteristics, and relative operating levels. Wea. Forecasting, 14, 713-725.

Meehl, G. Α., J. M. Arblaster, D. M. Lawrence, A. Seth, Ε. Κ. Schneider, Β. P. Kirtman, and D. Min, 2006: Monsoon regimes in the CCSM3. J. Climate, 19, 2482-2495.

Moore, A. M., and R. Kleeman, 1998: Skill assessment for ENSO using ensemble prediction. Quart. J. Roy. Meteor. Soc., 124, 557-584.

Morse, A. P., F. J. Doblas-Reyes, Μ. B. Hoshen, R. Hagedorn, and T. N. Palmer, 2005: A forecast quality assessment of an end-to-end probabilistic multi-model seasonal forecast system using a malaria model. Tellus, 57A, 464-475, doi:10.1111/ j.1600-0870.2005.00124.x.

Pacanowski, R. C., and S. M. Griffies, 1998: MOM 3.0 manual. NOAA/Geophysical Fluid Dynamics Laboratory, Princeton, NJ, 638 pp.

Palmer, T. N., C. Brankovic, and D. S. Richardson, 2000: A probability and decision-model analysis of PROVOST seasonal multimodel ensemble integrations. Quart. J. Roy. Meteor. Soc., 126, 2013-2034.

�, and Coauthors, 2004: Development of a European Multi-model Ensemble System for Seasonal-to-Interannual Prediction (DEMETER). Bull. Amer. Meteor. Soc., 85, 853-872.

�, F. J. Doblas-Reyes, A. Weisheimer, and M. J. Rodwell, 2008:

Toward seamless prediction: Calibration of climate change projections using seasonal forecast. Bull. Amer. Meteor. Soc., 89, 459-470.

Pope, V. D., M. L. Gallani, P. R. Rowntree, and R. A. Stratton, 2000: The impact of new physical parametrizations in the

Hadley Centre climate model�HadAM3. Climate Dyn., 16, 123-146.

Reynolds, R. W., and T. M. Smith, 1994: Improved global sea surface temperature analyses using optimum interpolation. J. Climate, 7, 929-948.

Rosati, Α., R. Gudgel, and K. Miyakoda, 1997: The impact of ocean initial conditions on ENSO forecasting with a coupled model. Mon. Wea. Rev., US, 754-772.

Saha, S., and Coauthors, 2006: The NCEP Climate Forecast System. J. Climate, 19, 3483-3517.

Schneider, Ε. Κ., B. Huang, Z. Zhu, D. G. DeWitt, J. L. Kinter III, B. Kirtman, and J. Shukla, 1999: Ocean data assimilation, initialization, and predictions of ENSO with a coupled GCM. Mon. Wea. Rev., 127, 1187-1207.

�, D. G. DeWitt, A. Rosati, B. P. Kirtman, L. Ji, and

J. J. Tribbia, 2003: Retrospective ENSO forecasts: Sensitivity to atmospheric model and ocean resolution. Mon. Wea. Rev., 131, 3038-3060.

Stan, C., and B. P. Kirtman, 2008: The influence of atmospheric noise and uncertainty in ocean initial conditions on the limit of predictability in a coupled GCM. J. Climate, 21, 3487-3503.

Stockdale, T. N., 1997: Coupled ocean-atmosphere forecasts in the presence of climate drift. J. Climate, 10, 809-818.

�, D. L. T. Anderson, J. O. S. Alves, and M. A. Balmaseda,

1998: Global seasonal rainfall forecasts using a coupled ocean-atmosphere model. Nature, 392, 370-373.

Van Oldenborgh, G. J., M. A. Balmaseda, L. Ferranti, T. N. Stockdale, and D. L. T. Anderson, 2005: Did the ECMWF seasonal forecast model outperform statistical ENSO forecast models over the last 15 years? J. Climate, 18, 3240-3249.

Vintzileos, Α., M. M. Rienecker, M. J. Suarez, S. D. Schubert, and S. Κ. Miller, 2005: Local versus remote wind forcing of the equatorial Pacific surface temperature in July 2003. Geophys. Res. Lett., 32, L05702, doi:10.1029/2004GL021972.

Wang, G., R. Kleeman, N. Smith, and F. Tseitkin, 2002: The BMRC coupled general circulation model ENSO forecast system. Mon. Wea. Rev., 130, 975-991.

Wang, W., S. Saha, H.-L. Pan, S. Nadiga, and G. White, 2005: Simulation of ENSO in the new NCEP coupled forecast system model. Mon. Wea. Rev., 133, 1574-1593.

Wu, R., B. P. Kirtman, and H. van den Dool, 2009: An analysis of ENSO prediction skill in the CFS retrospective forecasts. J. Climate, 22, 1801-1818.

Xue, Y., A. Leetmaa, and M. Ji, 2000: ENSO prediction with Markov models: The impact of sea level. J. Climate, 13, 849-871.

COPYRIGHT INFORMATION

Title: Multimodel Ensemble ENSO Prediction with CCSM and CFS Source: Mon Weather Rev 137 no9 S 2009 p. 2908-30 ISSN: 0027-0644 DOI: 10.1175/2009MWR2672.1 Publisher: American Meteorological Society 45 Beacon Street, Boston, MA 02108-3693 The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited. To contact the publisher: http://www.ametsoc.org/AMS/

This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.