Search Results

An important element in using evidence to select therapy is the determination of whether a treatment is clinically superior to its competitors. This chapter argues that the determination is a decision requiring both kinds of information that are necessary in decision analysis: the probabilities and values associated with the possible outcomes. It is debatable whether significance tests answer questions about probabilities in a form suitable for decision making. But significance tests cannot answer questions about the comparative values of different treatments. The preferable option is the one with the highest expected utility, where expected utility is the product of probability times utility.

Statistical significance testing can be viewed as a formal method for assessing the evidence with regard to a scientific conjecture. It reflects the degree to which we can be confident that an association observed in the study sample is also present in the target population. Significance tests play a central role in psychiatric epidemiology, and in fact across the full gamut of human research. Given the omnipresence of statistical significance tests, it is essential for all investigators to understand the principles behind these tests and guidelines for their interpretation. This chapter provides this background and makes recommendations for sensible evaluation of test results.

The standing of accounting among the academic disciplines has never been very high, in part because the work of accounting is not regarded as suitably creative. Yet when we think of knowledge as a set of institutionalized practices, the subservience of accounting to the sciences and to the ancient professions may be reversed. Fields like economics, engineering, biology, and medicine cannot escape the twin imperatives of commensuration and accountability, especially when these are brought to bear on matters of recognized public importance. In this regard, the career of cost-benefit quantification is exemplary. Knowledge and rationality, whenever they touch on politics and policy, have become closely bound up with a logic of accountancy.

This chapter discusses random error. Topics covered include taking a sequential approach to considering random and systematic error, special considerations in evaluating random error in observational studies, statistical significance testing, multiple comparisons and related issues, interpretation of confidence intervals, and integrated assessment of random error.

This chapter covers statistics from a conceptual rather than a quantitative perspective. The objective is to distinguish statistical from scientific hypotheses and provide background for later discussions. The fact that scientific and statistical hypotheses are dissimilar is often overlooked, and the chapter shows how they differ and why the differences matter. The chapter uses simple, nontechnical examples to present the main ideas in frequentist statistics; the next chapter covers the basics of Bayesian statistics and explores how the divergent philosophical viewpoints of frequentists and Bayesians affect scientific reasoning. The present chapter covers statistical error and its role in null hypothesis significance testing, which is still the dominant mode of scientific significance testing, as well as the origins, weaknesses of, and alternatives to, null-hypothesis testing. A philosophical objective of the chapter is to show how the concept of probability fits with Karl Popper’s program of falsification. The background material in this chapter will be especially important for evaluating the Reproducibility Crisis in Chapter 7.

This chapter defines and compares systematic reviews and meta-analyses. Systematic reviews carefully locate relevant evidence and appraise study qualities, while meta-analyses provide quantitative summaries of evidence, showing central trends, variations, and possible explanations for differences in results across studies. The advantages and disadvantages of these review methodologies are discussed, along with appropriate applications and quality standards.

Chapter 3 provides a brief overview of null hypothesis significance testing and points out its primary defects. It then outlines the neo-Fisherian account of tests of statistical significance, along with a second option contained in the philosophy of statistics known as the error-statistical philosophy, both of which are defensible. Tests of statistical significance are the most widely used means for evaluating hypotheses and theories in psychology. A massive critical literature has developed in psychology, and the behavioral sciences more generally, regarding the worth of these tests. The chapter provides a list of important lessons learned from the ongoing debates about tests of significance.

The chapter evaluates the well-known statistical-significance rule for discovering hypotheses and shows that, because scientists routinely misuse this rule, especially with observational and not experimental data, they can miss discovering important causal hypotheses in statistics. In particular, scientists forget how subjective the rule is, and that the rule tells nothing about scientific significance, only mathematical significance

This chapter discusses the epistemological scope of field science by exploring the econometric methodology of Trygve Haavelmo. The theories of social field phenomena are inexact; they do not provide the complete set of causes that affects the phenomenon. Haavelmo’s epistemology shows that a social field science explanation can never be made complete by statistics and statistical analysis only. In other words, a social field science cannot be a purely inductive science. Data observed “passively” might not display enough variations to reveal the relevant causal influences. Variation shows statistical significance, not potential influence. We also need theory. But because theory if fundamentally incomplete—inexact—we need additional sources of knowledge.

Milton Friedman’s empirical research is very different in tone and substance to the research in a typical journal article. This chapter points out that the difference in tone and substance is directly related to Friedman’s views on the foundations of statistics. Instead of viewing statistics as a classical statistician might, Friedman viewed statistics through the lens of personal probability. This had substantial implications for his overall research agenda and the particulars of his empirical work. Friedman emphasized agreement across various sets of data because such evidence was more likely to produce consensus among economists than tests on a single set of data. He put less emphasis on statistical significance of results, at least partly because of the problems involved in interpreting multiple tests on the same data.

This chapter describes a method for constructing confidence intervals for general equilibrium model results using the covariance matrices routinely computed in the course of estimating the model's parameters. The method is then applied to IGEM to produce confidence intervals for several types of results: (i) the levels of variables in IGEM's base case, (ii) deviations in variables resulting from a policy experiment consisting of a carbon tax used to reduce the rate of tax on capital income, and (iii) the effects of the carbon tax swap on the components of social welfare. For each case, a decomposition analysis is used to identify the key uncertainties underlying the confidence interval. The carbon tax swap is shown to produce statistically significant gains in key variables, including social welfare, along with significant reductions in pollution. The method is also shown to have important advantages over alternative approaches. The confidence intervals it produces are very similar to those generated by Monte Carlo simulation but are much faster to compute for large models. In addition, the decomposition analysis shows that off-diagonal terms in the parameter covariance matrices often contribute significantly to the confidence intervals for key results. By capturing these effects, the method provides better characterization of uncertainty than alternative methodologies that rely on independent perturbations of parameters.