Partial information decomposition

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.

Motivation

Information theory can quantify the amount of information a single source variable $X_{1}$ has about a target variable $Y$ via the mutual information $I(X_{1};Y)$. If we now consider a second source variable $X_{2}$, classical information theory can only describe the mutual information of the joint variable $\{X_{1},X_{2}\}$ with $Y$, given by $I(X_{1},X_{2};Y)$. In general however, it would be interesting to know how exactly the individual variables $X_{1}$ and $X_{2}$ and their interactions relate to $Y$.

Consider that we are given two source variables $X_{1},X_{2}\in \{0,1\}$ and a target variable $Y=XOR(X_{1},X_{2})$. In this case the total mutual information $I(X_{1},X_{2};Y)=1$, while the individual mutual information $I(X_{1};Y)=I(X_{2};Y)=0$. That is, there is synergistic information arising from the interaction of $X_{1},X_{2}$ about $Y$, which cannot be easily captured with classical information theoretic quantities.

Definition

Partial information decomposition further decomposes the mutual information between the source variables $\{X_{1},X_{2}\}$ with the target variable $Y$ as

$I(X_{1},X_{2};Y)={\text{Unq}}(X_{1};Y\setminus X_{2})+{\text{Unq}}(X_{2};Y\setminus X_{1})+{\text{Syn}}(X_{1},X_{2};Y)+{\text{Red}}(X_{1},X_{2};Y)$

Here the individual information atoms are defined as

• ${\text{Unq}}(X_{1};Y\setminus X_{2})$ is the unique information that $X_{1}$ has about $Y$, which is not in $X_{2}$
• ${\text{Syn}}(X_{1},X_{2};Y)$ is the synergistic information that is in the interaction of $X_{1}$ and $X_{2}$ about $Y$
• ${\text{Red}}(X_{1},X_{2};Y)$ is the redundant information that is in both $X_{1}$ or $X_{2}$ about $Y$

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.

Applications

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology, neuroscience sociology, and machine learning Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems and may be relevant to formal theories of consciousness.

See also

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information