Jan Modersitzki
- Published in print:
- 2003
- Published Online:
- September 2007
- ISBN:
- 9780198528418
- eISBN:
- 9780191713583
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198528418.003.0007
- Subject:
- Mathematics, Applied Mathematics
This chapter summarizes the techniques discussed so far in this book. The techniques are all based on the minimization of a certain distance measure, and the distance measure is based on image ...
More
This chapter summarizes the techniques discussed so far in this book. The techniques are all based on the minimization of a certain distance measure, and the distance measure is based on image features or directly on image intensities. Image features can be user supplied (e.g., landmarks) or may be deduced automatically from the image intensities (e.g., principal axes). Typical examples of intensity-based distance measures are the sum of squared differences, correlation or mutual information. For all proposed techniques, the transformation is parametric, i.e., it can be expanded in terms of some parameters and basis functions. The desired transformation is a minimizer of the distance measure in the space spanned by the basis functions. The minimizer can be obtained from algebraic equations or by applying appropriate optimization tools.Less
This chapter summarizes the techniques discussed so far in this book. The techniques are all based on the minimization of a certain distance measure, and the distance measure is based on image features or directly on image intensities. Image features can be user supplied (e.g., landmarks) or may be deduced automatically from the image intensities (e.g., principal axes). Typical examples of intensity-based distance measures are the sum of squared differences, correlation or mutual information. For all proposed techniques, the transformation is parametric, i.e., it can be expanded in terms of some parameters and basis functions. The desired transformation is a minimizer of the distance measure in the space spanned by the basis functions. The minimizer can be obtained from algebraic equations or by applying appropriate optimization tools.
Jan Modersitzki
- Published in print:
- 2003
- Published Online:
- September 2007
- ISBN:
- 9780198528418
- eISBN:
- 9780191713583
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198528418.003.0006
- Subject:
- Mathematics, Applied Mathematics
This chapter investigates the question of how to find an optimal linear transformation based on a distance measure. Popular choices for distance measures such as the sum of squared differences, ...
More
This chapter investigates the question of how to find an optimal linear transformation based on a distance measure. Popular choices for distance measures such as the sum of squared differences, correlation, and mutual information are discussed. Particular attention is paid to the differentiability of the distance measures. The desired transformation is restricted to a parameterizable space, and as such can be expanded in terms of a linear combination of some basis functions. The registration task is considered as an optimization problem, where the objective is to find the optimal coefficient in the expansion while minimizing the distance measure. The well-known Gauss-Newton method is described and used for numerical optimization. Different examples are used to identify similarities and differences of the distance measures.Less
This chapter investigates the question of how to find an optimal linear transformation based on a distance measure. Popular choices for distance measures such as the sum of squared differences, correlation, and mutual information are discussed. Particular attention is paid to the differentiability of the distance measures. The desired transformation is restricted to a parameterizable space, and as such can be expanded in terms of a linear combination of some basis functions. The registration task is considered as an optimization problem, where the objective is to find the optimal coefficient in the expansion while minimizing the distance measure. The well-known Gauss-Newton method is described and used for numerical optimization. Different examples are used to identify similarities and differences of the distance measures.
Gennaro Auletta
- Published in print:
- 2011
- Published Online:
- September 2011
- ISBN:
- 9780199608485
- eISBN:
- 9780191729539
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199608485.003.0003
- Subject:
- Physics, Soft Matter / Biological Physics
Here it is shown that quantum systems can be understood as information processors. Information and entropy are related quantities but also different, since the first is formal whilst the second is ...
More
Here it is shown that quantum systems can be understood as information processors. Information and entropy are related quantities but also different, since the first is formal whilst the second is dynamical. Both quantum and classical information acquisition are a three-step process that needs a processor, a regulator, and a decider.Less
Here it is shown that quantum systems can be understood as information processors. Information and entropy are related quantities but also different, since the first is formal whilst the second is dynamical. Both quantum and classical information acquisition are a three-step process that needs a processor, a regulator, and a decider.
Gennaro Auletta
- Published in print:
- 2011
- Published Online:
- September 2011
- ISBN:
- 9780199608485
- eISBN:
- 9780191729539
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199608485.003.0007
- Subject:
- Physics, Soft Matter / Biological Physics
In order to explain how the brain and also elementary organisms are able to refer to external things and processes we need to consider complexity. Complexity is a specific combination of order and ...
More
In order to explain how the brain and also elementary organisms are able to refer to external things and processes we need to consider complexity. Complexity is a specific combination of order and disorder in which several subsystems are interconnected but do not share an overall information. This allows for information encapsulation and modularization as well as for the necessary plasticity of organisms. A proto-metabolism can emerge when several autocatalytic processes are interconnected.Less
In order to explain how the brain and also elementary organisms are able to refer to external things and processes we need to consider complexity. Complexity is a specific combination of order and disorder in which several subsystems are interconnected but do not share an overall information. This allows for information encapsulation and modularization as well as for the necessary plasticity of organisms. A proto-metabolism can emerge when several autocatalytic processes are interconnected.
Kevin B. Korb, Erik P. Nyberg, and Lucas Hope
- Published in print:
- 2011
- Published Online:
- September 2011
- ISBN:
- 9780199574131
- eISBN:
- 9780191728921
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199574131.003.0030
- Subject:
- Mathematics, Logic / Computer Science / Mathematical Philosophy
The causal power of C over E is (roughly) the degree to which changes in C cause changes in E. A formal measure of causal power would be very useful, as an aid to understanding and modelling complex ...
More
The causal power of C over E is (roughly) the degree to which changes in C cause changes in E. A formal measure of causal power would be very useful, as an aid to understanding and modelling complex stochastic systems. Previous attempts to measure causal power, such as those of Good (1961), Cheng (1997), and Glymour (2001), while useful, suffer from one fundamental flaw: they only give sensible results when applied to very restricted types of causal system, all of which exhibit causal transitivity. Causal Bayesian networks, however, are not in general transitive. The chapter develops an information‐theoretic alternative, causal information, which applies to any kind of causal Bayesian network. Causal information is based upon three ideas. First, the chapter assumes that the system can be represented causally as a Bayesian network. Second, the chapter uses hypothetical interventions to select the causal from the non‐causal paths connecting C to E. Third, we use a variation on the information‐theoretic measure mutual information to summarize the total causal influence of C on E. The chapter's measure gives sensible results for a much wider variety of complex stochastic systems than previous attempts and promises to simplify the interpretation and application of Bayesian networks.Less
The causal power of C over E is (roughly) the degree to which changes in C cause changes in E. A formal measure of causal power would be very useful, as an aid to understanding and modelling complex stochastic systems. Previous attempts to measure causal power, such as those of Good (1961), Cheng (1997), and Glymour (2001), while useful, suffer from one fundamental flaw: they only give sensible results when applied to very restricted types of causal system, all of which exhibit causal transitivity. Causal Bayesian networks, however, are not in general transitive. The chapter develops an information‐theoretic alternative, causal information, which applies to any kind of causal Bayesian network. Causal information is based upon three ideas. First, the chapter assumes that the system can be represented causally as a Bayesian network. Second, the chapter uses hypothetical interventions to select the causal from the non‐causal paths connecting C to E. Third, we use a variation on the information‐theoretic measure mutual information to summarize the total causal influence of C on E. The chapter's measure gives sensible results for a much wider variety of complex stochastic systems than previous attempts and promises to simplify the interpretation and application of Bayesian networks.
Marc Mézard and Andrea Montanari
- Published in print:
- 2009
- Published Online:
- September 2009
- ISBN:
- 9780198570837
- eISBN:
- 9780191718755
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198570837.003.0001
- Subject:
- Physics, Theoretical, Computational, and Statistical Physics
This chapter introduces some of the basic concepts of information theory, as well as the definitions and notations of probability theory that are used throughout the book. It defines the fundamental ...
More
This chapter introduces some of the basic concepts of information theory, as well as the definitions and notations of probability theory that are used throughout the book. It defines the fundamental notions of entropy, relative entropy, and mutual information. It also presents the main questions of information theory: data compression and data transmission. Finally, it offers a brief introduction to error correcting codes and Shannon's theory.Less
This chapter introduces some of the basic concepts of information theory, as well as the definitions and notations of probability theory that are used throughout the book. It defines the fundamental notions of entropy, relative entropy, and mutual information. It also presents the main questions of information theory: data compression and data transmission. Finally, it offers a brief introduction to error correcting codes and Shannon's theory.
Vlatko Vedral
- Published in print:
- 2006
- Published Online:
- January 2010
- ISBN:
- 9780199215706
- eISBN:
- 9780191706783
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199215706.003.0005
- Subject:
- Physics, Theoretical, Computational, and Statistical Physics
Information is often considered classical in a definite state rather than in a superposition of states. It seems rather strange to consider information in superpositions. Some people would, on the ...
More
Information is often considered classical in a definite state rather than in a superposition of states. It seems rather strange to consider information in superpositions. Some people would, on the basis of this argument, conclude that quantum information can never exist and we can only have access to classical information. It turns out, however, that quantum information can be quantified in the same way as classical information using Shannon's prescription. There is a unique measure (up to a constant additive or multiplicative term) of quantum information such that S (the von Neumann entropy) is purely a function of the probabilities of outcomes of measurements made on a quantum system (that is, a function of a density operator); S is a continuous function of probability; S is additive. This chapter discusses the fidelity of pure quantum states, Helstrom's discrimination, quantum data compression, entropy of observation, conditional entropy and mutual information, relative entropy, and statistical interpretation of relative entropy.Less
Information is often considered classical in a definite state rather than in a superposition of states. It seems rather strange to consider information in superpositions. Some people would, on the basis of this argument, conclude that quantum information can never exist and we can only have access to classical information. It turns out, however, that quantum information can be quantified in the same way as classical information using Shannon's prescription. There is a unique measure (up to a constant additive or multiplicative term) of quantum information such that S (the von Neumann entropy) is purely a function of the probabilities of outcomes of measurements made on a quantum system (that is, a function of a density operator); S is a continuous function of probability; S is additive. This chapter discusses the fidelity of pure quantum states, Helstrom's discrimination, quantum data compression, entropy of observation, conditional entropy and mutual information, relative entropy, and statistical interpretation of relative entropy.
Jakob Hohwy
- Published in print:
- 2013
- Published Online:
- January 2014
- ISBN:
- 9780199682737
- eISBN:
- 9780191766350
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199682737.003.0003
- Subject:
- Philosophy, Philosophy of Mind
The central mechanism for hierarchical perceptual inference is prediction on the basis of internal, generative models, revision of model parameters, and minimization of prediction error. This is the ...
More
The central mechanism for hierarchical perceptual inference is prediction on the basis of internal, generative models, revision of model parameters, and minimization of prediction error. This is the way in which the brain engages in perceptual inference, as described in the previous chapter. This chapter describes this idea in detail. It uses a statistical analogy of model fitting to explain the notion of prediction error and then gradually builds up more and more complex versions of the theory, ending with broad ideas from information theory and statistical physics concerning mutual information, free energy and surprisal. The overall picture is of a self-supervised system that is closely supervised by the sensory signal it receives from the world, but which is hidden behind the veil of sensory input. This is a profound reversal of the way we normally think about the top-down and bottom-up signals in the brain. The system is able to recognize the causes of its sensory input in a mechanistic manner, by implicitly inverting its generative model.Less
The central mechanism for hierarchical perceptual inference is prediction on the basis of internal, generative models, revision of model parameters, and minimization of prediction error. This is the way in which the brain engages in perceptual inference, as described in the previous chapter. This chapter describes this idea in detail. It uses a statistical analogy of model fitting to explain the notion of prediction error and then gradually builds up more and more complex versions of the theory, ending with broad ideas from information theory and statistical physics concerning mutual information, free energy and surprisal. The overall picture is of a self-supervised system that is closely supervised by the sensory signal it receives from the world, but which is hidden behind the veil of sensory input. This is a profound reversal of the way we normally think about the top-down and bottom-up signals in the brain. The system is able to recognize the causes of its sensory input in a mechanistic manner, by implicitly inverting its generative model.
Masashi Sugiyama and Motoaki Kawanabe
- Published in print:
- 2012
- Published Online:
- September 2013
- ISBN:
- 9780262017091
- eISBN:
- 9780262301220
- Item type:
- chapter
- Publisher:
- The MIT Press
- DOI:
- 10.7551/mitpress/9780262017091.003.0011
- Subject:
- Computer Science, Machine Learning
This chapter summarizes the main themes covered in the preceding discussions and discusses future prospects. This book has provided a comprehensive overview of theory, algorithms, and applications of ...
More
This chapter summarizes the main themes covered in the preceding discussions and discusses future prospects. This book has provided a comprehensive overview of theory, algorithms, and applications of machine learning under covariate shift. Beyond covariate shift adaptation, it has been shown recently that the ratio of probability densities can be used for solving machine learning tasks. This novel machine learning framework includes multitask learning, privacy-preserving data mining, outlier detection, change detection in time series, two-sample test, conditional density estimation, and probabilistic classification. Furthermore, mutual information—which plays a central role in information theory—can be estimated via density ratio estimation.Less
This chapter summarizes the main themes covered in the preceding discussions and discusses future prospects. This book has provided a comprehensive overview of theory, algorithms, and applications of machine learning under covariate shift. Beyond covariate shift adaptation, it has been shown recently that the ratio of probability densities can be used for solving machine learning tasks. This novel machine learning framework includes multitask learning, privacy-preserving data mining, outlier detection, change detection in time series, two-sample test, conditional density estimation, and probabilistic classification. Furthermore, mutual information—which plays a central role in information theory—can be estimated via density ratio estimation.
Jakob Hohwy
- Published in print:
- 2013
- Published Online:
- January 2014
- ISBN:
- 9780199682737
- eISBN:
- 9780191766350
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199682737.003.0009
- Subject:
- Philosophy, Philosophy of Mind
This chapter deals with a series of philosophical problems. Primarily, it is the problem of misrepresentation, namely how an account of perceptual content can make room for the possibility of ...
More
This chapter deals with a series of philosophical problems. Primarily, it is the problem of misrepresentation, namely how an account of perceptual content can make room for the possibility of misperceiving the world. This is addressed in terms of average prediction error minimization, and by appeal to notions of mutual information. The chapter delves into deeper issues about the possibility of rule-following and briefly sets out the rather radical way in which the prediction error account, in terms of free energy and the link to statistical physics, can begin to approach this problem. The chapter then situates the prediction error account within the classic debates about representation, as having both causal elements and descriptive elements. After briefly suggesting how such an account might speak to the content of conscious experience, the chapter finally describes which kind of response it would entail for the famous Chinese room problem.Less
This chapter deals with a series of philosophical problems. Primarily, it is the problem of misrepresentation, namely how an account of perceptual content can make room for the possibility of misperceiving the world. This is addressed in terms of average prediction error minimization, and by appeal to notions of mutual information. The chapter delves into deeper issues about the possibility of rule-following and briefly sets out the rather radical way in which the prediction error account, in terms of free energy and the link to statistical physics, can begin to approach this problem. The chapter then situates the prediction error account within the classic debates about representation, as having both causal elements and descriptive elements. After briefly suggesting how such an account might speak to the content of conscious experience, the chapter finally describes which kind of response it would entail for the famous Chinese room problem.
Amos Golan
- Published in print:
- 2017
- Published Online:
- November 2017
- ISBN:
- 9780199349524
- eISBN:
- 9780199349555
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780199349524.003.0003
- Subject:
- Economics and Finance, Econometrics
In this chapter I present the key ideas and develop the essential quantitative metrics needed for modeling and inference with limited information. I provide the necessary tools to study the ...
More
In this chapter I present the key ideas and develop the essential quantitative metrics needed for modeling and inference with limited information. I provide the necessary tools to study the traditional maximum-entropy principle, which is the cornerstone for info-metrics. The chapter starts by defining the primary notions of information and entropy as they are related to probabilities and uncertainty. The unique properties of the entropy are explained. The derivations and discussion are extended to multivariable entropies and informational quantities. For completeness, I also discuss the complete list of the Shannon-Khinchin axioms behind the entropy measure. An additional derivation of information and entropy, due to the independently developed work of Wiener, is provided as well.Less
In this chapter I present the key ideas and develop the essential quantitative metrics needed for modeling and inference with limited information. I provide the necessary tools to study the traditional maximum-entropy principle, which is the cornerstone for info-metrics. The chapter starts by defining the primary notions of information and entropy as they are related to probabilities and uncertainty. The unique properties of the entropy are explained. The derivations and discussion are extended to multivariable entropies and informational quantities. For completeness, I also discuss the complete list of the Shannon-Khinchin axioms behind the entropy measure. An additional derivation of information and entropy, due to the independently developed work of Wiener, is provided as well.
Mark Newman
- Published in print:
- 2018
- Published Online:
- October 2018
- ISBN:
- 9780198805090
- eISBN:
- 9780191843235
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780198805090.003.0014
- Subject:
- Physics, Theoretical, Computational, and Statistical Physics
A discussion of community structure in networks and methods for its detection. The chapter begins with an introduction to the idea of community structure, followed by descriptions of a range of ...
More
A discussion of community structure in networks and methods for its detection. The chapter begins with an introduction to the idea of community structure, followed by descriptions of a range of methods for finding communities, including modularity maximization, the InfoMap method, methods based on maximum-likelihood fits of models to network data, betweenness-based methods, and hierarchical clustering. Also discussed are methods for assessing algorithm performance, along with a summary of performance studies and their findings. The chapter concludes with a discussion of other types of large-scale structure in networks, such as overlapping and hierarchical communities, core-periphery structure, latent-space structure, and rank structure.Less
A discussion of community structure in networks and methods for its detection. The chapter begins with an introduction to the idea of community structure, followed by descriptions of a range of methods for finding communities, including modularity maximization, the InfoMap method, methods based on maximum-likelihood fits of models to network data, betweenness-based methods, and hierarchical clustering. Also discussed are methods for assessing algorithm performance, along with a summary of performance studies and their findings. The chapter concludes with a discussion of other types of large-scale structure in networks, such as overlapping and hierarchical communities, core-periphery structure, latent-space structure, and rank structure.
Ginestra Bianconi
- Published in print:
- 2018
- Published Online:
- July 2018
- ISBN:
- 9780198753919
- eISBN:
- 9780191815676
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780198753919.003.0008
- Subject:
- Physics, Theoretical, Computational, and Statistical Physics
Multilayer networks have a mesoscale structure organized in multilayer communities, spanning different layers and often revealing important functional properties of the network. In this chapter the ...
More
Multilayer networks have a mesoscale structure organized in multilayer communities, spanning different layers and often revealing important functional properties of the network. In this chapter the major techniques proposed for detecting and characterizing the multilayer communities are described, including: generalized modularity, consensus clustering, multilayer infomaps, multilink communities, tensorial decomposition, Normalized Mutual Information, theta indicators. The main benefits and limitations of these approaches are discussed and revealed by analysing the results obtained on real datasets coming from sociology, technology, molecular biology and brain networks. Additionally, techniques for layer aggregation and disaggregation are here discussed. These methods are compared and commented in order to provide a general perspective on the subject.Less
Multilayer networks have a mesoscale structure organized in multilayer communities, spanning different layers and often revealing important functional properties of the network. In this chapter the major techniques proposed for detecting and characterizing the multilayer communities are described, including: generalized modularity, consensus clustering, multilayer infomaps, multilink communities, tensorial decomposition, Normalized Mutual Information, theta indicators. The main benefits and limitations of these approaches are discussed and revealed by analysing the results obtained on real datasets coming from sociology, technology, molecular biology and brain networks. Additionally, techniques for layer aggregation and disaggregation are here discussed. These methods are compared and commented in order to provide a general perspective on the subject.
