Search Results

Model identification is a necessary component of modern science. Model misspecification is a major, if not the dominant, source of error in the quantification of most scientific evidence. Hypothesis tests have become the de facto standard for evidence in the bulk of scientific work. This chapter discusses the information criteria approach to model identification, which can be thought of as an extension of the likelihood ratio approach to the case of multiple alternatives. It shows that the information criteria approach can be extended to large sets of statistical models. There is a tradeoff between the amount of model detail that can be accurately captured and the number of models that can be considered. This tradeoff can be incorporated in modifications of the parameter penalty term. The chapter also examines the Akaike information criterion and its variants, such as Schwarz's information criterion. It demonstrates how a data-based penalty can be developed to take into account the working model complexity, model set complexity, and sample size.

The likelihood principle has been defended on Bayesian grounds, with proponents insisting that it coincides with and systematizes intuitive judgments about example problems, and that it generalizes what is true when hypotheses have deductive consequences about observations. Richard Royall offers three kinds of justification. He points out, first, that the likelihood principle makes intuitive sense when probabilities are all 1s and 0s. His second argument is that the likelihood ratio is precisely the factor that transforms a ratio of prior probabilities into a ratio of posteriors. His third line of defense of the likelihood principle is to show that it coincides with intuitive judgments about evidence when the principle is applied to specific cases. This chapter divides the principle into two parts—one qualitative, the other quantitative—and evaluates each in the light of the Akaike information criterion (AIC). Both turn out to be correct in a special case (when the competing hypotheses have the same number of adjustable parameters), but not otherwise.

This chapter addresses the problem of model selection. The success of machine learning techniques depends heavily on the choice of hyperparameters such as basis functions, the kernel bandwidth, the regularization parameter, and the importance-flattening parameter. Thus, model selection is one of the most fundamental and crucial topics in machine learning. Standard model selection schemes such as the Akaike information criterion, cross-validation, and the subspace information criterion have their own theoretical justification in terms of the unbiasedness as generalization error estimators. However, such theoretical guarantees are no longer valid under covariate shift. The chapter introduces their modified variants using importance-weighting techniques, and shows that the modified methods are properly unbiased even under covariate shift. The usefulness of these modified model selection criteria is illustrated through numerical experiments.

This chapter describes the pruning algorithm for calculating the likelihood on a tree, as well as extensions under complex substitution models, including the gamma and covarion models of rate variation among sites and lineages. It discusses numerical optimization algorithms for maximum likelihood estimation. It provides a critical assessment of methods for reconstructing ancestral states for both molecular sequences and morphological characters. Finally the chapter discusses model selection in phylogenetics using the likelihood ratio test (LRT) and information criteria such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC).

There are several additional statistical procedures that can be conducted after a habitat analysis. The statistical model produced by a habitat analysis can be assessed for fit to the data. Model fit describes how well the predictor variables explain the variance in the response variable, typically species presence–absence or abundance. When more than one statistical model has been produced by the habitat analysis, these can be compared by a formal procedure called model comparison. This usually involves identifying the model with the lowest Akaike information criterion (AIC) value. If the statistical model is considered a predictive tool then its predictive accuracy needs to be assessed. There are many metrics for assessing the predictive performance of a model and quantifying rates of correct and incorrect classification; the latter are error rates. Many of these metrics are based on the numbers of true positive, true negative, false positive, and false negative observations in an independent dataset. “True” and “false” refer to whether species presence–absence was correctly predicted or not. Predictive performance can also be assessed by constructing a receiver operating characteristic (ROC) curve and calculating area under the curve (AUC) values. High AUC values approaching 1 indicate good predictive performance, whereas a value near 0.5 indicates a poor model that predicts species presence–absence no better than a random guess.