Biometrika Advance Access originally published online on February 6, 2008
Biometrika 2008 95(1):169-186; doi:10.1093/biomet/asm086
Articles |
Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models
Department of Statistics, Warwick University, Coventry, CV4 7AL, U.K. O.Papaspiliopoulos{at}warwick.ac.uk Gareth.O.Roberts{at}warwick.ac.uk
Received for publication 1 April 2004. Revision received 1 June 2007.
Inference for Dirichlet process hierarchical models is typically performed using Markov chain Monte Carlo methods, which can be roughly categorized into marginal and conditional methods. The former integrate out analytically the infinite-dimensional component of the hierarchical model and sample from the marginal distribution of the remaining variables using the Gibbs sampler. Conditional methods impute the Dirichlet process and update it as a component of the Gibbs sampler. Since this requires imputation of an infinite-dimensional process, implementation of the conditional method has relied on finite approximations. In this paper, we show how to avoid such approximations by designing two novel Markov chain Monte Carlo algorithms which sample from the exact posterior distribution of quantities of interest. The approximations are avoided by the new technique of retrospective sampling. We also show how the algorithms can obtain samples from functionals of the Dirichlet process. The marginal and the conditional methods are compared and a careful simulation study is included, which involves a non-conjugate model, different datasets and prior specifications.
Key Words: Exact simulation • Label switching • Mixture model • Retrospective sampling • Stick-breaking prior
References
-
Antoniak C. E. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. (1974) 2:1152–74.[CrossRef]
Beskos A., Papaspiliopoulos O., Roberts G. O. Retrospective exact simulation of diffusion sample paths with applications. Bernoulli (2006) 12:1077–98.
Beskos A., Papaspiliopoulos O., Roberts G. O., Fearnhead P. Exact and efficient likelihood–based inference for discretely observed diffusions (with Discussion). J. R. Statist. Soc. B (2006) 68:333–82.[CrossRef]
Beskos A., Papaspiliopoulos O., Roberts G. O. A new factorisation of diffusion measure and finite sample path constructions. Methodol. Comp. Appl. Prob. (2008) to appear.
Blackwell D., MacQueen J. B. Ferguson distributions via Pólya urn schemes. Ann. Statist. (1973) 1:353–5.[CrossRef]
Burr D., Doss H. A Bayesian semiparametric model for random-effects meta-analysis. J. Am. Statist. Assoc. (2005) 100:242–51.[CrossRef][Web of Science]
Cifarelli D. M., Regazzini E. Distribution functions of means of a Dirichlet process. Ann. Statist. (1990) 18:429–42.[CrossRef]
Doss H. Bayesian nonparametric estimation for incomplete data via successive substitution sampling. Ann. Statist. (1994) 22:1763–86.[CrossRef]
Dunson D., Park J. Kernel stick-breaking processes. Biometrika (2008) to appear.
Fearnhead P. Particle filters for mixture models with an unknown number of components. Statist. Comp. (2004) 14:11–21.[CrossRef]
Ferguson T. S. A Bayesian analysis of some nonparametric problems. Ann. Statist. (1973) 1:209–30.[CrossRef]
Ferguson T. S. Prior distributions on spaces of probability measures. Ann. Statist. (1974) 2:615–29.[CrossRef]
Ferguson T. S. Bayesian density estimation by mixtures of normal distributions. In: Recent Advances in Statistics: Papers in Honor of Herman Chernoff on His Sixtieth Birthday—Chernoff H., Haseeb M., Rustagi Rizvi J., Siegmund D., eds. (1983) New York: Academic Press. 287–302.
Gelfand A., Kottas A. Bayesian semiparametric regression for median residual life. Scand. J. Statist. (2003) 30:651–65.[CrossRef]
Gelfand A. E., Kottas A. A computational approach for full nonparametric Bayesian inference under Dirichlet process mixture models. J. Comput. Graph. Statist. (2002) 11:289–305.[CrossRef]
Green P., Richardson S. Modelling heterogeneity with and without the Dirichlet process. Scand. J. Statist. (2001) 28:355–75.[CrossRef]
Guglielmi A., Holmes C. C., Walker S. G. Perfect simulation involving functionals of a Dirichlet process. J. Comp. Graph. Statist. (2002) 11:306–10.[CrossRef]
Guglielmi A., Tweedie R. L. Markov chain Monte Carlo estimation of the law of the mean of a Dirichlet process. Bernoulli (2001) 7:573–92.[CrossRef]
Ishwaran H., James L. Gibbs sampling methods for stick-breaking priors. J. Am. Statist. Assoc. (2001) 96:161–73.[CrossRef][Web of Science]
Ishwaran H., James L. F. Some further developments for stick-breaking priors: finite and infinite clustering and classification. Sankhy
A (2003) 65:577–92.
Ishwaran H., Zarepour M. Markov chain Monte Carlo in approximate Dirichlet and beta two-parameter process hierarchical models. Biometrika (2000) 87:371–90.
Ishwaran H., Zarepour M. Exact and approximate sum-representations for the dirichlet process. Can. J. Statist. (2002) 30:269–83.[CrossRef]
Jain S., Neal R. M. A split-merge Markov chain Monte Carlo procedure for the Dirichlet process mixture model. J. Comp. Graph. Statist. (2004) 13:158–82.[CrossRef]
Liu J. S. The collapsed Gibbs sampler in Bayesian computations with applications to a gene regulation problem. J. Am. Statist. Assoc (1994) 89:958–66.[CrossRef][Web of Science]
Liu J. S. Nonparametric hierarchical Bayes via sequential imputations. Ann. Statist. (1996) 24:911–30.[CrossRef]
Lo A. Y. On a class of Bayesian nonparametric estimates. I. Density estimates. Ann. Statist. (1984) 12:351–7.[CrossRef]
MacEachern S., Müller P. Estimating mixture of Dirichlet process models. J. Comp. Graph. Statist. (1998) 7:223–38.[CrossRef]
Mengersen K. L., Tweedie R. L. Rates of convergence of the Hastings and Metropolis algorithms. Ann. Statist. (1996) 24:101–21.[CrossRef]
Müller P., Erkanli A., West M. Bayesian curve fitting using multivariate normal mixtures. Biometrika (1996) 83:67–79.
Müler P., Rosner G. L., De Iorio M., MacEachern S. A nonparametric Bayesian model for inference in related longitudinal studies. Appl. Statist. (2005) 54:611–26.
Neal R. Markov chain sampling: Methods for Dirichlet process mixture models. J. Comp. Graph. Statist. (2000) 9:283–97.
Papaspiliopoulos O., Roberts G. O., Sköld M. Non-centered parameterisations for hierarchical models and data augmentation. In: Bayesian Statistics 7—Bernardo J., Bayarri M., Berger J., Dawid A., Heckerman D., Smith A., West M., eds. (2003) Oxford: Oxford University Press. 307–27.
Papaspiliopoulos O., Roberts G. O., Sköld M. A general framework for parametrisation of hierarchical models. Statist. Sci. (2006) to appear.
Porteous I., Ihler A., Smyth P., Welling M. Gibbs sampling for (coupled) infinite mixture models in the stick breaking representation. In: Proceedings of the 22nd Annual Conference on Uncertainty in Artificial Intelligence (2006) Arlington, VA: AUAI Press. (UAI-06).
Quintana F., Iglesias P. Bayesian clustering and product partition models. J. R. Statist. Soc. B (2003) 65:557–74.[CrossRef]
Richardson S., Green P. J. On Bayesian analysis of mixtures with an unknown number of components (with Discussion). J. R. Statist. Soc. B (1997) 59:731–92.[CrossRef]
Ripley B. D. Stochastic Simulation (1987) Chichester: Wiley.
Roberts G. O. Markov chain concepts related to sampling algortihms. In: MCMC in Practice—Gilks W., Richardson S., Spiegelhalter D., eds. (1996) London: Chapman and Hall. 45–57.
Sethuraman J. A constructive definition of Dirichlet priors. Statist. Sinica (1994) 4:639–50.
Sokal A. Monte Carlo methods in statistical mechanics: Foundations and new algorithms. Functional Integration (Cargèse, 1996) (1997) vol. 361, of NATO Adv. Sci. Inst. Ser. B Phys. pp. 131–92, New York: Plenum.
Teh Y., Jordan M., Beal M., Blei D. Hierarchical dirichlet processes. J. Am. Statist. Assoc. (2006) 101:1566–81.[CrossRef]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. B. Dunson Nonparametric Bayes local partition models for random effects Biometrika, June 1, 2009; 96(2): 249 - 262. [Abstract] [PDF]
|
|
|
|
|
|
D. B. Dunson and J.-H. Park Kernel stick-breaking processes Biometrika, June 1, 2008; 95(2): 307 - 323. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||