Reducing Model Selection Computational Cost by Metamodeling the Evidence

摘要

Modeling is an abstraction of reality that lets a designer or engineer perform simulations with a "real" object or structure without testing it physically. However, the result of simulations is as good as their models. Selecting the best model among many candidates is another challenge.;Complex models could translate into more accurate predictions, but they also require larger computational effort. The Bayes factor is often used to compare different models considering the data and analyst's experience. To obtain the Bayes factor, the Bayesian model evidence is calculated; however, calculating the Bayesian model evidence of computationally expensive models is not a straightforward process. The Bayesian evidence is obtained by integrating the multiplication of the likelihood and prior, which is usually performed by numerical methods like Monte Carlo integration algorithms. However, to apply Monte Carlo integration many model evaluations are required, and this is not always feasible when the model is computationally expensive. Metamodeling techniques may be used to reduce the computational cost by replacing the full model with a metamodel. In this research, a different approach to what is proposed in the literature is proposed, and the metamodeling is applied to model the integrand of the Bayesian evidence. Moreover, Probability Distribution Functions, PDFs, are proposed for the metamodel because the integrand of the Bayesian evidence shares some common characteristics with PDFs. For example, they are both always positive. The hypervolume defined by the integrand function is not the same as the hypervolume under the PDF. Therefore to fit the PDF to the integrand, in addition to the PDF parameters, the use of a scale factor is proposed. This scale factor is used to estimate the Bayesian evidence. To fit the PDFs to the integrand, a scale factor is needed that finally is used to estimate the Bayesian evidence. The proposed method is explained using several examples. It is shown that the number of samples needed is significantly less than the number of samples when Monte Carlo integration methods are used.