Biometrika Advance Access originally published online on January 30, 2009
Biometrika 2009 96(1):175-186; doi:10.1093/biomet/asn068
Articles |
Reducing variability of crossvalidation for smoothing-parameter choice
Department of Mathematics and Statistics, The University of Melbourne, Victoria 3010, Australia halpstat{at}ms.unimelb.edu.au andrewpr{at}ms.unimelb.edu.au
Received for publication 1 August 2007. Revision received 1 June 2008.
One of the attractions of crossvalidation, as a tool for smoothing-parameter choice, is its applicability to a wide variety of estimator types and contexts. However, its detractors comment adversely on the relatively high variance of crossvalidatory smoothing parameters, noting that this compromises the performance of the estimators in which those parameters are used. We show that the variability can be reduced simply, significantly and reliably by employing bootstrap aggregation or bagging. We establish that in theory, when bagging is implemented using an adaptively chosen resample size, the variability of crossvalidation can be reduced by an order of magnitude. However, it is arguably more attractive to use a simpler approach, based for example on half-sample bagging, which can reduce variability by approximately 50%.
Key Words: Bagging • Bandwidth • Bootstrap aggregation • Half-sampling • Kernel estimation • Nonparametric density estimation • Nonparametric regression • Statistical smoothing • Subsampling
References
-
Borra S., Di Ciaccio A. Performance evaluation of bagging and boosting in nonparametric regression. Metron (2001) 59:141–56.
Borra S., Di Ciaccio A. Improving nonparametric regression methods by bagging and boosting. Comput. Statist. Data Anal. (2002) 38:407–20.[CrossRef]
Bowman A. W. An alternative method of cross-validation for the smoothing of density estimates. Biometrika (1984) 71:353–60.
Breiman L. Bagging predictors. Mach. Learn. (1996) 24:123–40.
Bühlmann P. Bagging, boosting and ensemble methods. In: Handbook of Computational Statistics—Gentle J. E., Härdle W., Mori Y., eds. (2004) Berlin: Springer. 877–907.
Bühlmann P., Yu B. Analyzing bagging. Ann. Statist. (2002) 30:927–61.[CrossRef]
Buja A., Stuetzle W. Observations on bagging. Statist. Sinica (2006) 16:323–51.
Cao R., Cuevas A., González-Manteiga W. A comparative study of several smoothing methods in density estimation. Comput. Statist. Data Anal. (1994) 17:153–76.[CrossRef]
Chen S. X., Hall P. Effects of bagging and bias correction on estimators defined by estimating equations. Statist. Sinica (2003) 13:97–109.
Chiu S. T. Bandwidth selection for kernel density estimation. Ann. Statist. (1991) 19:1883–905.[CrossRef]
Craven P., Wahba G. Smoothing noisy data with spline functions. Estimating the correct degree of smoothing by the method of generalized cross-validation. Numer. Math. (1978) 31:377–403.[CrossRef]
Delaigle A., Hall P. Using SIMEX for smoothing-parameter choice in errors-in-variables problems. J. Am. Statist. Assoc. (2008) 103:280–7.[CrossRef][Web of Science]
Dey A. K., Ruymgaart F. H., Mair B. A. Cross-validation for parameter selection in inverse estimation problems. Scand. J. Statist. (1996) 23:609–20.
Fan J. Local linear regression estimators and their minimax efficiencies. Ann. Statist. (1993) 21:196–216.[CrossRef]
Faraway J. J., Jhun M. Bootstrap choice of bandwidth for density estimation. J. Am. Statist. Assoc. (1990) 85:1119–22.[CrossRef][Web of Science]
Feluch W., Koronacki J. A note on modified cross-validation in density estimation. Comp. Statist. Data Anal. (1992) 13:143–51.[CrossRef]
Friedman J. H., Hall P. On bagging and nonlinear estimation. J. Statist. Plan. Infer. (2007) 137:669–83.[CrossRef]
Hall P. Asymptotic properties of integrated square error and cross-validation for kernel estimation of a regression function. Z. Wahr. verw. Geb. (1984) 67:175–96.[CrossRef]
Hall P., Johnstone I. M. Empirical functional and efficient smoothing parameter selection. J. R. Statist. Soc. (1992) B 54:475–530.
Hall P., Marron J. S. Extent to which least-squares cross-validation minimises integrated square error in nonparametric density estimation. Prob. Theory Rel. Fields (1987) 74:567–81.[CrossRef]
Hall P., Marron J. S., Park B. U. Smoothed cross-validation. Prob. Theory Rel. Fields (1992) 92:1–20.[CrossRef]
Hall P., Samworth R. J. Properties of bagged nearest neighbour classifiers. J. R. Statist. Soc. (2005) B 67:363–79.[CrossRef]
Hall P., Sheather J., Jones M. C., Marron J. S. On optimal data-based bandwidth selection in kernel density estimation. Biometrika (1991) 78:263–9.
Härdle W., Hall P., Marron J. S. How far are automatically chosen regression smoothing parameters from their optimum? J. Am. Statist. Assoc. (1988) 83:86–101.[CrossRef][Web of Science]
Härdle W., Hall P., Marron J. S. Regression smoothing parameters that are not far from their optimum. J. Am. Statist. Assoc. (1992) 87:227–33.[CrossRef][Web of Science]
Hesse C. H. Data-driven deconvolution. Nonparam. Statist. (1999) 10:343–73.[CrossRef]
Hothorn T., Lausen B. Bundling classifiers by bagging trees. Comp. Statist. Data Anal. (2005) 49:1068–78.[CrossRef]
Jones M. C., Marron J. S., Park B. U. A simple root n bandwidth selector. Ann. Statist. (1991) 19:1919–32.[CrossRef]
Kauermann G., Opsomer J. D. Generalized cross-validation for bandwidth selection of backfitting estimates in generalized additive models. J. Comp. Graph. Statist. (2004) 13:66–89.[CrossRef]
Loader C. R. Bandwidth selection: Classical or plug-in? Ann. Statist. (1999) 27:415–38.
Lukas M. A. Robust generalized cross-validation for choosing the regularization parameter. Inverse Problems (2006) 22:1883–902.[CrossRef][Web of Science]
Marron J. S., Wand M. P. Exact mean-integrated squared error. Ann. Statist. (1992) 20:712–36.[CrossRef]
Park B. U., Marron J. S. Comparison of data-driven bandwidth selectors. J. Am. Statist. Assoc. (1990) 85:66–72.[CrossRef][Web of Science]
R Development Core Team. R:. In: A Language and Environment for Statistical Computing (2008) Vienna: R Foundation for Statistical Computing.
Rice J. Bandwidth choice for nonparametric regression. Ann. Statist. (1984) 12:1215–30.[CrossRef]
Ridgeway G. Looking for lumps: Boosting and bagging for density estimation. Comp. Statist. Data Anal. (2002) 38:379–92.[CrossRef]
Rudemo M. Empirical choice of histograms and kernel density estimators. Scand. J. Statist. (1982) 9:65–78.
Scott D. W., Terrell G. R. Biased and unbiased cross-validation in density estimation. J. Am. Statist. Assoc. (1987) 82:1131–46.[CrossRef][Web of Science]
Sinisi S. E., Neugebauer R., Van Der Laan M. J. Cross-validated bagged prediction of survival. Statist. Appl. Genet. Molec. Biol. (2006) 5:26.
Stute W. Modified cross-validation in density estimation. J. Statist. Plan. Infer. (1992) 30:293–305.[CrossRef]
Taylor C. C. Bootstrap choice of the smoothing parameter in kernel density estimation. Biometrika (1989) 76:705–12.
Wahba G., Wold S. A completely automatic French curve: Fitting spline functions by cross validation. Commun. Statist. (1975) 4:1–17.
Wand M. P., Jones M. C. Multivariate plug-in bandwidth selection. Comp. Statist. (1994) 9:97–116.
Wong W. H. On the consistency of cross-validation in kernel nonparametric regression. Ann. Statist. (1983) 11:1136–41.
Wu C. O. A cross-validation bandwidth choice for kernel density estimates with selection biased data. J. Mult. Anal. (1997) 61:38–60.[CrossRef]
Zhang H. Classification trees for multiple binary responses. J. Am. Statist. Assoc. (1998) 93:180–93.[CrossRef][Web of Science]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||