Mark L. Taper
- Published in print:
- 2004
- Published Online:
- February 2013
- ISBN:
- 9780226789552
- eISBN:
- 9780226789583
- Item type:
- chapter
- Publisher:
- University of Chicago Press
- DOI:
- 10.7208/chicago/9780226789583.003.0015
- Subject:
- Biology, Ecology
Model identification is a necessary component of modern science. Model misspecification is a major, if not the dominant, source of error in the quantification of most scientific evidence. Hypothesis ...
More
Model identification is a necessary component of modern science. Model misspecification is a major, if not the dominant, source of error in the quantification of most scientific evidence. Hypothesis tests have become the de facto standard for evidence in the bulk of scientific work. This chapter discusses the information criteria approach to model identification, which can be thought of as an extension of the likelihood ratio approach to the case of multiple alternatives. It shows that the information criteria approach can be extended to large sets of statistical models. There is a tradeoff between the amount of model detail that can be accurately captured and the number of models that can be considered. This tradeoff can be incorporated in modifications of the parameter penalty term. The chapter also examines the Akaike information criterion and its variants, such as Schwarz's information criterion. It demonstrates how a data-based penalty can be developed to take into account the working model complexity, model set complexity, and sample size.Less
Model identification is a necessary component of modern science. Model misspecification is a major, if not the dominant, source of error in the quantification of most scientific evidence. Hypothesis tests have become the de facto standard for evidence in the bulk of scientific work. This chapter discusses the information criteria approach to model identification, which can be thought of as an extension of the likelihood ratio approach to the case of multiple alternatives. It shows that the information criteria approach can be extended to large sets of statistical models. There is a tradeoff between the amount of model detail that can be accurately captured and the number of models that can be considered. This tradeoff can be incorporated in modifications of the parameter penalty term. The chapter also examines the Akaike information criterion and its variants, such as Schwarz's information criterion. It demonstrates how a data-based penalty can be developed to take into account the working model complexity, model set complexity, and sample size.
Malcolm Forster and Elliott Sober
- Published in print:
- 2004
- Published Online:
- February 2013
- ISBN:
- 9780226789552
- eISBN:
- 9780226789583
- Item type:
- chapter
- Publisher:
- University of Chicago Press
- DOI:
- 10.7208/chicago/9780226789583.003.0006
- Subject:
- Biology, Ecology
The likelihood principle has been defended on Bayesian grounds, with proponents insisting that it coincides with and systematizes intuitive judgments about example problems, and that it generalizes ...
More
The likelihood principle has been defended on Bayesian grounds, with proponents insisting that it coincides with and systematizes intuitive judgments about example problems, and that it generalizes what is true when hypotheses have deductive consequences about observations. Richard Royall offers three kinds of justification. He points out, first, that the likelihood principle makes intuitive sense when probabilities are all 1s and 0s. His second argument is that the likelihood ratio is precisely the factor that transforms a ratio of prior probabilities into a ratio of posteriors. His third line of defense of the likelihood principle is to show that it coincides with intuitive judgments about evidence when the principle is applied to specific cases. This chapter divides the principle into two parts—one qualitative, the other quantitative—and evaluates each in the light of the Akaike information criterion (AIC). Both turn out to be correct in a special case (when the competing hypotheses have the same number of adjustable parameters), but not otherwise.Less
The likelihood principle has been defended on Bayesian grounds, with proponents insisting that it coincides with and systematizes intuitive judgments about example problems, and that it generalizes what is true when hypotheses have deductive consequences about observations. Richard Royall offers three kinds of justification. He points out, first, that the likelihood principle makes intuitive sense when probabilities are all 1s and 0s. His second argument is that the likelihood ratio is precisely the factor that transforms a ratio of prior probabilities into a ratio of posteriors. His third line of defense of the likelihood principle is to show that it coincides with intuitive judgments about evidence when the principle is applied to specific cases. This chapter divides the principle into two parts—one qualitative, the other quantitative—and evaluates each in the light of the Akaike information criterion (AIC). Both turn out to be correct in a special case (when the competing hypotheses have the same number of adjustable parameters), but not otherwise.
David F. Hendry
- Published in print:
- 2014
- Published Online:
- January 2015
- ISBN:
- 9780262028356
- eISBN:
- 9780262324410
- Item type:
- chapter
- Publisher:
- The MIT Press
- DOI:
- 10.7551/mitpress/9780262028356.003.0017
- Subject:
- Economics and Finance, Econometrics
There are many possible methods of model selection, most of which can be implemented in an automatic algorithm. General contenders include single path approaches such as forward and backward ...
More
There are many possible methods of model selection, most of which can be implemented in an automatic algorithm. General contenders include single path approaches such as forward and backward selection, mixed variants like step-wise, multi-path search methods including Hoover and Perez (1999) and PcGets, information criteria (AIC, BIC etc.), Lasso, and RETINA, as well as a number of selection algorithms specifically designed for a forecasting context (such as PIC: see e.g., Phillips, 1995, 1996, and Phillips and Ploberger, 1996). Here we only consider the former group of methods in relation to Autometrics, partly to evaluate improvements over time in the conventional selection aspects of its performance. Three key findings are that Autometrics does indeed deliver substantive improvements in many settings; that performance is not necessarily adversely affected by having more variables than observations; and that although other approaches sometimes outperform, they are not reliable and can also deliver very poor results, whereas Autometrics tends to perform similarly to commencing from the LDGP using the same significance level.Less
There are many possible methods of model selection, most of which can be implemented in an automatic algorithm. General contenders include single path approaches such as forward and backward selection, mixed variants like step-wise, multi-path search methods including Hoover and Perez (1999) and PcGets, information criteria (AIC, BIC etc.), Lasso, and RETINA, as well as a number of selection algorithms specifically designed for a forecasting context (such as PIC: see e.g., Phillips, 1995, 1996, and Phillips and Ploberger, 1996). Here we only consider the former group of methods in relation to Autometrics, partly to evaluate improvements over time in the conventional selection aspects of its performance. Three key findings are that Autometrics does indeed deliver substantive improvements in many settings; that performance is not necessarily adversely affected by having more variables than observations; and that although other approaches sometimes outperform, they are not reliable and can also deliver very poor results, whereas Autometrics tends to perform similarly to commencing from the LDGP using the same significance level.
Andy Hector
- Published in print:
- 2015
- Published Online:
- March 2015
- ISBN:
- 9780198729051
- eISBN:
- 9780191795855
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198729051.003.0011
- Subject:
- Biology, Biomathematics / Statistics and Data Analysis / Complexity Studies, Ecology
Generalized linear mixed-effects models (GLMMs) are introduced as a combination of the mixed-effects models and GLMs met in earlier chapters. The additional challenges of the analysis are explored ...
More
Generalized linear mixed-effects models (GLMMs) are introduced as a combination of the mixed-effects models and GLMs met in earlier chapters. The additional challenges of the analysis are explored and the differences in the R software functions and its output are explained. Model comparison and selection can be done using information criteria or likelihood ratio tests. In addition to the AIC and BIC the DIC was devised for use with multilevel models. The options for assessing how well the model assumptions are met are reduced relative to linear models and GLMs but the most accessible current options are demonstrated.Less
Generalized linear mixed-effects models (GLMMs) are introduced as a combination of the mixed-effects models and GLMs met in earlier chapters. The additional challenges of the analysis are explored and the differences in the R software functions and its output are explained. Model comparison and selection can be done using information criteria or likelihood ratio tests. In addition to the AIC and BIC the DIC was devised for use with multilevel models. The options for assessing how well the model assumptions are met are reduced relative to linear models and GLMs but the most accessible current options are demonstrated.
Masashi Sugiyama and Motoaki Kawanabe
- Published in print:
- 2012
- Published Online:
- September 2013
- ISBN:
- 9780262017091
- eISBN:
- 9780262301220
- Item type:
- chapter
- Publisher:
- The MIT Press
- DOI:
- 10.7551/mitpress/9780262017091.003.0003
- Subject:
- Computer Science, Machine Learning
This chapter addresses the problem of model selection. The success of machine learning techniques depends heavily on the choice of hyperparameters such as basis functions, the kernel bandwidth, the ...
More
This chapter addresses the problem of model selection. The success of machine learning techniques depends heavily on the choice of hyperparameters such as basis functions, the kernel bandwidth, the regularization parameter, and the importance-flattening parameter. Thus, model selection is one of the most fundamental and crucial topics in machine learning. Standard model selection schemes such as the Akaike information criterion, cross-validation, and the subspace information criterion have their own theoretical justification in terms of the unbiasedness as generalization error estimators. However, such theoretical guarantees are no longer valid under covariate shift. The chapter introduces their modified variants using importance-weighting techniques, and shows that the modified methods are properly unbiased even under covariate shift. The usefulness of these modified model selection criteria is illustrated through numerical experiments.Less
This chapter addresses the problem of model selection. The success of machine learning techniques depends heavily on the choice of hyperparameters such as basis functions, the kernel bandwidth, the regularization parameter, and the importance-flattening parameter. Thus, model selection is one of the most fundamental and crucial topics in machine learning. Standard model selection schemes such as the Akaike information criterion, cross-validation, and the subspace information criterion have their own theoretical justification in terms of the unbiasedness as generalization error estimators. However, such theoretical guarantees are no longer valid under covariate shift. The chapter introduces their modified variants using importance-weighting techniques, and shows that the modified methods are properly unbiased even under covariate shift. The usefulness of these modified model selection criteria is illustrated through numerical experiments.
Ziheng Yang
- Published in print:
- 2014
- Published Online:
- August 2014
- ISBN:
- 9780199602605
- eISBN:
- 9780191782251
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199602605.003.0004
- Subject:
- Biology, Biomathematics / Statistics and Data Analysis / Complexity Studies, Evolutionary Biology / Genetics
This chapter describes the pruning algorithm for calculating the likelihood on a tree, as well as extensions under complex substitution models, including the gamma and covarion models of rate ...
More
This chapter describes the pruning algorithm for calculating the likelihood on a tree, as well as extensions under complex substitution models, including the gamma and covarion models of rate variation among sites and lineages. It discusses numerical optimization algorithms for maximum likelihood estimation. It provides a critical assessment of methods for reconstructing ancestral states for both molecular sequences and morphological characters. Finally the chapter discusses model selection in phylogenetics using the likelihood ratio test (LRT) and information criteria such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC).Less
This chapter describes the pruning algorithm for calculating the likelihood on a tree, as well as extensions under complex substitution models, including the gamma and covarion models of rate variation among sites and lineages. It discusses numerical optimization algorithms for maximum likelihood estimation. It provides a critical assessment of methods for reconstructing ancestral states for both molecular sequences and morphological characters. Finally the chapter discusses model selection in phylogenetics using the likelihood ratio test (LRT) and information criteria such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC).
Bert Timmermans and Axel Cleeremans
- Published in print:
- 2015
- Published Online:
- June 2015
- ISBN:
- 9780199688890
- eISBN:
- 9780191801785
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199688890.003.0003
- Subject:
- Psychology, Cognitive Psychology, Developmental Psychology
The study of consciousness requires a solution to the following fundamental problem: How can we measure consciousness? While there has been substantial progress in measuring the level of awareness, ...
More
The study of consciousness requires a solution to the following fundamental problem: How can we measure consciousness? While there has been substantial progress in measuring the level of awareness, and researchers have made steady progress delineating the neural correlates of consciousness, it remains impossible to measure the contents of awareness directly and link a person’s subjective experience to an objective state of the world or the person. In a historical overview this chapter highlights how research has moved from a strict dissociation logic and the search for subjective and objective thresholds for awareness to more graded approaches in which conscious and unconscious processes are recognized to contribute to all measurements. Subsequently, it delineates the most important challenges to behavioral methods such as exhaustiveness, exclusiveness, and sensitivity. In particular it focuses on the risk of confounding awareness and metacognition, and the tension between exclusiveness of a measure and the information criterion.Less
The study of consciousness requires a solution to the following fundamental problem: How can we measure consciousness? While there has been substantial progress in measuring the level of awareness, and researchers have made steady progress delineating the neural correlates of consciousness, it remains impossible to measure the contents of awareness directly and link a person’s subjective experience to an objective state of the world or the person. In a historical overview this chapter highlights how research has moved from a strict dissociation logic and the search for subjective and objective thresholds for awareness to more graded approaches in which conscious and unconscious processes are recognized to contribute to all measurements. Subsequently, it delineates the most important challenges to behavioral methods such as exhaustiveness, exclusiveness, and sensitivity. In particular it focuses on the risk of confounding awareness and metacognition, and the tension between exclusiveness of a measure and the information criterion.
Jan Sprenger and Stephan Hartmann
- Published in print:
- 2019
- Published Online:
- October 2019
- ISBN:
- 9780199672110
- eISBN:
- 9780191881671
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780199672110.003.0010
- Subject:
- Philosophy, Philosophy of Science
Is simplicity a virtue of a good scientific theory, and are simpler theories more likely to be true or predictively successful? If so, how much should simplicity count vis-à-vis predictive accuracy? ...
More
Is simplicity a virtue of a good scientific theory, and are simpler theories more likely to be true or predictively successful? If so, how much should simplicity count vis-à-vis predictive accuracy? We address this question using Bayesian inference, focusing on the context of statistical model selection and an interpretation of simplicity via the degree of freedoms of a model. We rebut claims to prove the epistemic value of simplicity by means of showing its particular role in Bayesian model selection strategies (e.g., the BIC or the MML). Instead, we show that Bayesian inference in the context of model selection is usually done in a philosophically eclectic, instrumental fashion that is more tuned to practical applications than to philosophical foundations. Thus, these techniques cannot justify a particular “appropriate weight of simplicity in model selection”.Less
Is simplicity a virtue of a good scientific theory, and are simpler theories more likely to be true or predictively successful? If so, how much should simplicity count vis-à-vis predictive accuracy? We address this question using Bayesian inference, focusing on the context of statistical model selection and an interpretation of simplicity via the degree of freedoms of a model. We rebut claims to prove the epistemic value of simplicity by means of showing its particular role in Bayesian model selection strategies (e.g., the BIC or the MML). Instead, we show that Bayesian inference in the context of model selection is usually done in a philosophically eclectic, instrumental fashion that is more tuned to practical applications than to philosophical foundations. Thus, these techniques cannot justify a particular “appropriate weight of simplicity in model selection”.
Andy Hector
- Published in print:
- 2015
- Published Online:
- March 2015
- ISBN:
- 9780198729051
- eISBN:
- 9780191795855
- Item type:
- book
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198729051.001.0001
- Subject:
- Biology, Biomathematics / Statistics and Data Analysis / Complexity Studies, Ecology
Statistics is a fundamental component of the scientific toolbox, but learning the basics of this area of mathematics is one of the most challenging parts of a research training. This book gives an ...
More
Statistics is a fundamental component of the scientific toolbox, but learning the basics of this area of mathematics is one of the most challenging parts of a research training. This book gives an up-to-date introduction to the classical techniques and modern extensions of linear model analysis—one of the most useful approaches in the analysis of scientific data in the life and environmental sciences. The book emphasizes an estimation-based approach that takes account of recent criticisms of over-use of probability values and introduces the alternative approach using information criteria. The book is based on the use of the open-source R programming language for statistics and graphics that is rapidly becoming the lingua franca in many areas of science. Statistics is introduced through worked analyses performed in R using interesting data sets from ecology, evolutionary biology, and environmental science. The data sets and R scripts are available as supporting material.Less
Statistics is a fundamental component of the scientific toolbox, but learning the basics of this area of mathematics is one of the most challenging parts of a research training. This book gives an up-to-date introduction to the classical techniques and modern extensions of linear model analysis—one of the most useful approaches in the analysis of scientific data in the life and environmental sciences. The book emphasizes an estimation-based approach that takes account of recent criticisms of over-use of probability values and introduces the alternative approach using information criteria. The book is based on the use of the open-source R programming language for statistics and graphics that is rapidly becoming the lingua franca in many areas of science. Statistics is introduced through worked analyses performed in R using interesting data sets from ecology, evolutionary biology, and environmental science. The data sets and R scripts are available as supporting material.
Therese M. Donovan and Ruth M. Mickey
- Published in print:
- 2019
- Published Online:
- July 2019
- ISBN:
- 9780198841296
- eISBN:
- 9780191876820
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780198841296.003.0018
- Subject:
- Biology, Biomathematics / Statistics and Data Analysis / Complexity Studies
This chapter provides a very brief introduction to Bayesian model selection. The “Survivor Problem” is expanded in this chapter, where the focus is now on comparing two models that predict how long a ...
More
This chapter provides a very brief introduction to Bayesian model selection. The “Survivor Problem” is expanded in this chapter, where the focus is now on comparing two models that predict how long a contestant will last in a game of Survivor: one model uses years of formal education as a predictor, and a second model uses grit as a predictor. Gibbs sampling is used for parameter estimation. Deviance Information Criterion (commonly abbreviated as DIC) is used as a guide for model selection. Details of how this measure is computed are described. The chapter also discusses model assessment (model fit) and Occam’s razor.Less
This chapter provides a very brief introduction to Bayesian model selection. The “Survivor Problem” is expanded in this chapter, where the focus is now on comparing two models that predict how long a contestant will last in a game of Survivor: one model uses years of formal education as a predictor, and a second model uses grit as a predictor. Gibbs sampling is used for parameter estimation. Deviance Information Criterion (commonly abbreviated as DIC) is used as a guide for model selection. Details of how this measure is computed are described. The chapter also discusses model assessment (model fit) and Occam’s razor.
Joseph A. Veech
- Published in print:
- 2021
- Published Online:
- February 2021
- ISBN:
- 9780198829287
- eISBN:
- 9780191868078
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780198829287.003.0010
- Subject:
- Biology, Ecology, Biomathematics / Statistics and Data Analysis / Complexity Studies
There are several additional statistical procedures that can be conducted after a habitat analysis. The statistical model produced by a habitat analysis can be assessed for fit to the data. Model fit ...
More
There are several additional statistical procedures that can be conducted after a habitat analysis. The statistical model produced by a habitat analysis can be assessed for fit to the data. Model fit describes how well the predictor variables explain the variance in the response variable, typically species presence–absence or abundance. When more than one statistical model has been produced by the habitat analysis, these can be compared by a formal procedure called model comparison. This usually involves identifying the model with the lowest Akaike information criterion (AIC) value. If the statistical model is considered a predictive tool then its predictive accuracy needs to be assessed. There are many metrics for assessing the predictive performance of a model and quantifying rates of correct and incorrect classification; the latter are error rates. Many of these metrics are based on the numbers of true positive, true negative, false positive, and false negative observations in an independent dataset. “True” and “false” refer to whether species presence–absence was correctly predicted or not. Predictive performance can also be assessed by constructing a receiver operating characteristic (ROC) curve and calculating area under the curve (AUC) values. High AUC values approaching 1 indicate good predictive performance, whereas a value near 0.5 indicates a poor model that predicts species presence–absence no better than a random guess.Less
There are several additional statistical procedures that can be conducted after a habitat analysis. The statistical model produced by a habitat analysis can be assessed for fit to the data. Model fit describes how well the predictor variables explain the variance in the response variable, typically species presence–absence or abundance. When more than one statistical model has been produced by the habitat analysis, these can be compared by a formal procedure called model comparison. This usually involves identifying the model with the lowest Akaike information criterion (AIC) value. If the statistical model is considered a predictive tool then its predictive accuracy needs to be assessed. There are many metrics for assessing the predictive performance of a model and quantifying rates of correct and incorrect classification; the latter are error rates. Many of these metrics are based on the numbers of true positive, true negative, false positive, and false negative observations in an independent dataset. “True” and “false” refer to whether species presence–absence was correctly predicted or not. Predictive performance can also be assessed by constructing a receiver operating characteristic (ROC) curve and calculating area under the curve (AUC) values. High AUC values approaching 1 indicate good predictive performance, whereas a value near 0.5 indicates a poor model that predicts species presence–absence no better than a random guess.