An Algorithm for Computing Average Mutual Information using Probability Distribution Smoothing

Abstract

There is continuing interest in using Average Mutual Information (AMI) to quantify the pair-wise distance between dataset profiles. Among several algorithms used to find a numerical estimation of AMI, the histogram method is the most common since it provides simplicity and least cost. However, this algorithm is known to underestimate the computed entropies and to overestimate the resulting AMI.  Kernel Density Estimator (KDE)-based algorithms advanced to alleviate such systematic errors rely on bin-level smoothing. In the present work, we propose an alternative algorithm that uses smoothing on the probability distribution level. We consider several smoothing functions, both in the probability space and in its frequency space. An experimental approach is used to investigate the effect of such modification on the computation of both the entropy and the AMI. Results show that, to a significant extent, the present method is able to remove systematic errors in computing entropy and AMI. It is also shown that the present algorithm leads to better reconstruction of multivariate time series when AMI is used in conjunction with their independent components.