Mutual Information calculation using scikit-learn for a Bi-variate Gaussian; derivation of Analytical solution.

In this blog post, we will learn to calculate the Exact mutual information between two variables of a bivariate gaussian—the bi-variate gaussian looks like this.

2-D gaussian

Let's look at this distribution's Probability Density Function(PDF).

To sample from this distribution, we will use NumPy,

import numpy as np
import matplotlib.pyplot as plt
from numpy.random import multivariate_normal

We need to specify the mean and covariance matrix,

In the above snippet, we have constructed the covariance matrix and sampled 5000 data points from the 2-D normal distribution.

Now that we know what our data looks like, we need to calculate the Mutual Information b/w the two variables (X & Y).

Mutual Information:

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables.

It is a measure of the shared information b/w two variables; mutual information zero means two variables are mutually exclusive, while mutual info of infinity means two variables are identical.

Now that we know the formula for calculating mutual information let's go ahead and try to solve for our 2-D gaussian case.

for rho = 0.8, the mutual information value is 0.5108.

Now test the mutual information with the scikit-learn function.

In the above code, we are importing the mutual info library and drawing samples from the gaussian with varying sample sizes. We will see how the mutual information converges to the values we calculate in the previous step.

convergence of mutual information to Analytical solution

That all! Give a thumbs up if you benefited from this post.