Search Results

This chapter provides an introduction to large deviation theory. It begins with an overview of the motivatio n for the problem under study, focusing on probability distributions and how to construct an empirical distribution. It then considers the notion of a lower semi-continuous function and that of a lower semi-continuous relaxation before discussing the large deviation property for i.i.d. samples. In particular, it describes Sanov's theorem for a finite alphabet and proceeds by analyzing large deviation property for Markov chains, taking into account stationary distributions, entropy and relative entropy rates, the rate function for doubleton frequencies, and the rate function for singleton frequencies.

This chapter begins with the discussion of various meanings of the entropy: Boltzmann entropy, Shannon entropy, von Neumann entropy. It then presents the basic properties of the von Neumann entropy such as concavity, sub-additivity, strong sub-additivity, and continuity. It proves the existence of mean entropy for shift-invariant states on quantum spin chains, and then derives the expression for the mean entropy for quasi-free Fermions on a chain. The chapter defines the quantum relative entropy and presents its basic properties, including behaviour with respect to completely positive maps. The maximum entropy principle defines thermal equilibrium states (Gibbs states). This variational principle is illustrated by the Hartree–Fock approximation for a model of interacting Fermions. The entropy of an equilibrium state for a free quantum particle on a compact Riemannian manifold is also estimated. Finally, the notion of relative entropy is formulated in the algebraic setting using the relative modular operator.

This chapter provides an introduction to some elementary aspects of information theory, including entropy in its various forms. Entropy refers to the level of uncertainty associated with a random variable (or more precisely, the probability distribution of the random variable). When there are two or more random variables, it is worthwhile to study the conditional entropy of one random variable with respect to another. The last concept is relative entropy, also known as the Kullback–Leibler divergence, which measures the “disparity” between two probability distributions. The chapter first considers convex and concave functions before discussing the properties of the entropy function, conditional entropy, uniqueness of the entropy function, and the Kullback–Leibler divergence.

Information is often considered classical in a definite state rather than in a superposition of states. It seems rather strange to consider information in superpositions. Some people would, on the basis of this argument, conclude that quantum information can never exist and we can only have access to classical information. It turns out, however, that quantum information can be quantified in the same way as classical information using Shannon's prescription. There is a unique measure (up to a constant additive or multiplicative term) of quantum information such that S (the von Neumann entropy) is purely a function of the probabilities of outcomes of measurements made on a quantum system (that is, a function of a density operator); S is a continuous function of probability; S is additive. This chapter discusses the fidelity of pure quantum states, Helstrom's discrimination, quantum data compression, entropy of observation, conditional entropy and mutual information, relative entropy, and statistical interpretation of relative entropy.

This chapter discusses how entangled two states are. One obvious way is to consider the distillation procedure, in which the entanglement of any state is quantified by how many fully entangled qubits can be distilled from an entangled state. However, it is not easy to see how to distil singlets out of n copies of a general mixed state. Hence, it is necessary to first see what can be done with pure states. This chapter discusses the distillation of multiple copies of a pure state, analogy between quantum entanglement and the Carnot cycle, properties of entanglement measures, entanglement of pure states, entanglement of mixed states, measures of entanglement derived from relative entropy, relationship between classical information and entanglement, and links between entanglement and thermodynamics.

This chapter discusses the principles of quantum information. The quantum mechanical equivalent of the Shannon noisy-channel communication theorem is obtained, along with some profound statements about the behavior of quantum information during generalised quantum measurements. The latter are very important in studies of quantum entanglement, but also in exploring the connections between thermodynamics, information theory, and quantum physics. This chapter also discusses equalities and inequalities related to entropy, the Holevo bound, capacity of a bosonic channel, information gained through measurements, relative entropy and thermodynamics, entropy increase due to erasure, and Landauer's erasure and data compression.

In this chapter I present the key ideas and develop the essential quantitative metrics needed for modeling and inference with limited information. I provide the necessary tools to study the traditional maximum-entropy principle, which is the cornerstone for info-metrics. The chapter starts by defining the primary notions of information and entropy as they are related to probabilities and uncertainty. The unique properties of the entropy are explained. The derivations and discussion are extended to multivariable entropies and informational quantities. For completeness, I also discuss the complete list of the Shannon-Khinchin axioms behind the entropy measure. An additional derivation of information and entropy, due to the independently developed work of Wiener, is provided as well.

In this chapter I introduce and quantify prior information and show how to incorporate it into the info-metrics framework. The priors developed arise from fundamental properties of the system, from logical reasoning, or from empirical observations. I start the chapter with the derivation of priors for discrete distributions, which can be handled via the grouping property, and a detailed derivation of surprisal analysis. Constructing priors for continuous distributions is more challenging. That problem is tackled via the method of transformation groups, which is related to the mathematical concept of group theory. That method works for both discrete and continuous functions. The last approaches I discuss are based on empirical information. The close relationship between priors, treatment effects, and score functions is discussed and demonstrated in the last section. Visual illustrations of the theory and numerous theoretical and applied examples are provided.

Large deviations theory formulated by Varadhan has made a tremendous impact in a variety of fields such as mathematical physics, control theory, and statistics, to name a few. After a brief discussion of the general theory and examples, the large deviations principle (LDP) is shown to be equivalent to the Laplace principle in our context. The rate function for the LVP is obtained, in general, via relative entropy. Next, the Boué-Dupuis representation theorem for positive functionals of a Wiener process is established. Using the representation theorem, the Laplace principle is proved for diffusions.

The chapter introduces the concept of fitness, and focuses on the distribution of traits or behaviors within each population. The traits may be biological and governed by genes, or may instead have a social or a virtual basis that can be described by “memes.” Evolution describes how the distribution changes over time in response to fitness differences, for example, according to the replicator equation in discrete time or continuous time. The chapter also presents the Hardy‐Weinberg model of diploid evolution. A technical appendix points out connections to relative entropy and explores the continuous time limit of discrete replicator dynamics.

This chapter shows how replicator dynamics (in a setting with no frequency dependence) correspond to multiplicative updates studied by computer scientists in the context of online learning. The updates learn very quickly (a “blessing”) but they also wipe out potentially valuable variety that may be important when the environment changes (the “curse”). The chapter presents several different techniques developed in machine learning to lift the curse, and also different techniques seen in Nature. The chapter asks what machine learning can learn from Nature and vice versa when the recent past may be a treacherous guide to future events, as exemplified in the disk spin-down problem. Highlights include a demonstration of logical connections between Bayesian updating and replicator dynamics, and a discussion of how in-vitro selection techniques relate to computer algorithms that preserve diversity.

The traditional way of analyzing brain electrical activity, on the basis of electroencephalogram (EEG) records, relies mainly on visual inspection and years of training. Although it is quite useful, of course, one has to acknowledge its subjective nature that hardly allows for a systematic protocol. In order to overcome this undesirable feature, a quantitative EEG analysis has been developed over the years that introduces objective measures. These reflect not only characteristics of the brain activity itself, but also clues concerning the underlying associated neural dynamics. The processing of information by the brain is reflected in dynamical changes of the electrical activity in (i) time, (ii) frequency, and (iii) space. Therefore, the concomitant studies require methods capable of describing the qualitative variation of the signal in both time and frequency. In the present work we introduce new information tools based on the wavelet transform for the assessment of EEG data. In particular, different complexity measures are utilized…. The traditional electroencephalogram (EEG) tracing is now interpreted in much the same way as it was 50 years ago. More channels are used now and much more is known about clinical implication of the waves, but the basic EEG display and quantification of it are quite similar to those of its predecessors. The clinical interpretation of EEG records is made by a complex process of visual pattern recognition and the association with external and evident characteristics of clinical symptomatology. Analysis of EEG signals always involves the queries of quantification, i.e., the ability to state objective data in numerical and/or graphic form that simplify the analysis of long EEG time series. Without such measures, EEG appraisal remains subjective and can hardly lead to logical systematization [36]. Spectral decomposition of the EEG by computing the Fourier transform has been used since the very early days of electroencephalography. The rhythmic nature of many EEG activities lends itself naturally to this analysis. Fourier transform allows separation of various rhythms and estimation of their frequencies independently of each other, a difficult task to perform visually if several rhythmic activities occur simultaneously. Spectral analysis can also quantify the amount of activity in a frequency band.