Search Results

Model identification is a necessary component of modern science. Model misspecification is a major, if not the dominant, source of error in the quantification of most scientific evidence. Hypothesis tests have become the de facto standard for evidence in the bulk of scientific work. This chapter discusses the information criteria approach to model identification, which can be thought of as an extension of the likelihood ratio approach to the case of multiple alternatives. It shows that the information criteria approach can be extended to large sets of statistical models. There is a tradeoff between the amount of model detail that can be accurately captured and the number of models that can be considered. This tradeoff can be incorporated in modifications of the parameter penalty term. The chapter also examines the Akaike information criterion and its variants, such as Schwarz's information criterion. It demonstrates how a data-based penalty can be developed to take into account the working model complexity, model set complexity, and sample size.

The likelihood principle has been defended on Bayesian grounds, with proponents insisting that it coincides with and systematizes intuitive judgments about example problems, and that it generalizes what is true when hypotheses have deductive consequences about observations. Richard Royall offers three kinds of justification. He points out, first, that the likelihood principle makes intuitive sense when probabilities are all 1s and 0s. His second argument is that the likelihood ratio is precisely the factor that transforms a ratio of prior probabilities into a ratio of posteriors. His third line of defense of the likelihood principle is to show that it coincides with intuitive judgments about evidence when the principle is applied to specific cases. This chapter divides the principle into two parts—one qualitative, the other quantitative—and evaluates each in the light of the Akaike information criterion (AIC). Both turn out to be correct in a special case (when the competing hypotheses have the same number of adjustable parameters), but not otherwise.

Generalized linear mixed-effects models (GLMMs) are introduced as a combination of the mixed-effects models and GLMs met in earlier chapters. The additional challenges of the analysis are explored and the differences in the R software functions and its output are explained. Model comparison and selection can be done using information criteria or likelihood ratio tests. In addition to the AIC and BIC the DIC was devised for use with multilevel models. The options for assessing how well the model assumptions are met are reduced relative to linear models and GLMs but the most accessible current options are demonstrated.

This chapter addresses the problem of model selection. The success of machine learning techniques depends heavily on the choice of hyperparameters such as basis functions, the kernel bandwidth, the regularization parameter, and the importance-flattening parameter. Thus, model selection is one of the most fundamental and crucial topics in machine learning. Standard model selection schemes such as the Akaike information criterion, cross-validation, and the subspace information criterion have their own theoretical justification in terms of the unbiasedness as generalization error estimators. However, such theoretical guarantees are no longer valid under covariate shift. The chapter introduces their modified variants using importance-weighting techniques, and shows that the modified methods are properly unbiased even under covariate shift. The usefulness of these modified model selection criteria is illustrated through numerical experiments.

This chapter describes the pruning algorithm for calculating the likelihood on a tree, as well as extensions under complex substitution models, including the gamma and covarion models of rate variation among sites and lineages. It discusses numerical optimization algorithms for maximum likelihood estimation. It provides a critical assessment of methods for reconstructing ancestral states for both molecular sequences and morphological characters. Finally the chapter discusses model selection in phylogenetics using the likelihood ratio test (LRT) and information criteria such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC).

The study of consciousness requires a solution to the following fundamental problem: How can we measure consciousness? While there has been substantial progress in measuring the level of awareness, and researchers have made steady progress delineating the neural correlates of consciousness, it remains impossible to measure the contents of awareness directly and link a person’s subjective experience to an objective state of the world or the person. In a historical overview this chapter highlights how research has moved from a strict dissociation logic and the search for subjective and objective thresholds for awareness to more graded approaches in which conscious and unconscious processes are recognized to contribute to all measurements. Subsequently, it delineates the most important challenges to behavioral methods such as exhaustiveness, exclusiveness, and sensitivity. In particular it focuses on the risk of confounding awareness and metacognition, and the tension between exclusiveness of a measure and the information criterion.

Is simplicity a virtue of a good scientific theory, and are simpler theories more likely to be true or predictively successful? If so, how much should simplicity count vis-à-vis predictive accuracy? We address this question using Bayesian inference, focusing on the context of statistical model selection and an interpretation of simplicity via the degree of freedoms of a model. We rebut claims to prove the epistemic value of simplicity by means of showing its particular role in Bayesian model selection strategies (e.g., the BIC or the MML). Instead, we show that Bayesian inference in the context of model selection is usually done in a philosophically eclectic, instrumental fashion that is more tuned to practical applications than to philosophical foundations. Thus, these techniques cannot justify a particular “appropriate weight of simplicity in model selection”.

This chapter provides a very brief introduction to Bayesian model selection. The “Survivor Problem” is expanded in this chapter, where the focus is now on comparing two models that predict how long a contestant will last in a game of Survivor: one model uses years of formal education as a predictor, and a second model uses grit as a predictor. Gibbs sampling is used for parameter estimation. Deviance Information Criterion (commonly abbreviated as DIC) is used as a guide for model selection. Details of how this measure is computed are described. The chapter also discusses model assessment (model fit) and Occam’s razor.

There are several additional statistical procedures that can be conducted after a habitat analysis. The statistical model produced by a habitat analysis can be assessed for fit to the data. Model fit describes how well the predictor variables explain the variance in the response variable, typically species presence–absence or abundance. When more than one statistical model has been produced by the habitat analysis, these can be compared by a formal procedure called model comparison. This usually involves identifying the model with the lowest Akaike information criterion (AIC) value. If the statistical model is considered a predictive tool then its predictive accuracy needs to be assessed. There are many metrics for assessing the predictive performance of a model and quantifying rates of correct and incorrect classification; the latter are error rates. Many of these metrics are based on the numbers of true positive, true negative, false positive, and false negative observations in an independent dataset. “True” and “false” refer to whether species presence–absence was correctly predicted or not. Predictive performance can also be assessed by constructing a receiver operating characteristic (ROC) curve and calculating area under the curve (AUC) values. High AUC values approaching 1 indicate good predictive performance, whereas a value near 0.5 indicates a poor model that predicts species presence–absence no better than a random guess.