1. Introduction

Cao et al. ^[4] reviewed techniques for extracting local features for automatic object recognition in images. Multivariate Gaussians can represent the important features and their mutual correlations needed for accurate document retrieval from databases. The natural choice for discrimination between pairs of such distributions is the Fisher information metric on the Riemannian manifold of smooth probability density functions coordinatized by the parameters of the distribution ^[1,2]. However, it is not known analytically in some important cases of practical interest.

We have used multivariate Gaussians for face recognition using the neighbourhoods of colour pixel features at landmark points in face images ^[12], where we found that the spatial covariances among pixel colours was important. Craciunesco and Murari et al ^[5,8] used geodesic distance on Gaussian manifolds to interpret time series in very large databases from Tokomak measurements in fusion research. Verdoolaege, Shabbir et al ^[10,13] used multivariate generalized Gaussians for colour texture discrimination in the wavelet domain. In these studies the discrimination used approximations to the information distance between pairs of multivariate Gaussian probability density functions. Nielsen et al ^[9] suggested an entropic quantization method for approximating distances in the case of mixtures of multivariate Gaussians.

The $k$-variate Gaussian distributions have probability density functions:

$f({\mu}, {\Sigma}) = \frac{e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}}{\sqrt{{(2 \pi)}^k |\Sigma|}},$

(1)

where $x\in\mathbb{R}^k$ is the random variable, $\mu\in\mathbb{R}^k$ a $k$-dimensional mean vector, and $\Sigma\in\mathbb{R}^{(k^2+k)/2}$ is the $k \times k$ positive definite symmetric covariance matrix, for features with $k$-dimensional representation.

The Riemannian manifold $M^k$ of the family of $k$-variate Gaussians for a given $k$ is well understood through information geometric study using the Fisher metric ^[1,3,6,11]. For an introduction to information geometry and a range of applications see ^[1,2,7]. What we have analytically are natural metrics, on the space of means and on the space of covariances, giving the information distance between two multivariate Gaussians $f^A(\mu^A, \Sigma), f^B(\mu^B, \Sigma)$ of the same number $k$ of variables in two particular cases:

${\bf 1. \ \Sigma^A = \Sigma^B = \Sigma:}$ $f^A(\mu^A, \Sigma), f^B(\mu^B, \Sigma)$

Here we have the positive definite symmetric quadratic form $\Sigma$ to give a distance between two mean vectors:

$D_\mu(f^A, f^B) = \sqrt{\left(\mu ^A-\mu ^B\right)^T\cdot\Sigma^{-1} \cdot \left(\mu ^A-\mu^B\right)}.$

(2)

So also we have a norm on mean vectors for each $f^A(\mu^A, \Sigma):$

$|| \mu^A|| = \sqrt{(\mu^A)^T\cdot(\Sigma^A)^{-1} \cdot (\mu^A)} \ $

(3)

which is evidently sensitive to the covariance.

${\bf 2. \ \mu ^A = \mu ^B = \mu:}$ $f^A(\mu, \Sigma^A), f^B(\mu, \Sigma^B)$

Here we use a positive definite symmetric matrix constructed from $\Sigma^A$ and $\Sigma^B$ to give distance between two covariance matrices; this information metric was given by Atkinson and Mitchell ^[3] from a result attributed to S.T. Jensen, using

$S^{AB} = {\Sigma^A}^{-1/2} \cdot \Sigma^B \cdot {\Sigma^A}^{-1/2}, \ \ {\rm with \ } \ \{\lambda^{AB}_j\} = {\rm Eig} (S^{AB}) \ \ {\rm then}$

$\begin{eqnarray}\label{DdSigma} D_\Sigma(f^A, f^B) & = & \sqrt{\frac{1}{2} \sum\limits_{j = 1}^k \log^2(\lambda^{AB}_j)} . \end{eqnarray}$

(4)

We note that (4) is in agreement also with a special case of the geodesic distance given by Shabbir et al ^[10] for generalized multivariate Gaussians with the same mean.

In principle, (4) yields all of the geodesic distances since the information metric is invariant under affine transformations of the mean ^[3] Appendix 1; see also the article of P.S. Eriksen ^[6]. The equations for the geodesics were shown by Skovgaard ^[11] to be

$\begin{eqnarray}\label{geod} \ddot{\mu} & = & \dot{\Sigma} \Sigma^{-1} \dot{\mu} \nonumber \\ \ddot{\Sigma} & = & \dot{\Sigma} \Sigma^{-1} \dot{\Sigma} -\dot{\mu}\dot{\mu}^T . \end{eqnarray}$

(5)

Eriksen ^[6] observed that the family $M^k$ of $k$-variate Gaussians is isometric to the space $GA^+(k)/ SO(k)$ where $GA^+(k)$ consists of positive affine transformations. Hence by a translation it is sufficient to restrict the geodesic to one through $\Sigma = I$ the identity, in the direction $(-B, v).$ Then, through the change of coordinates, $\Delta = \Sigma^{-1}, \ \delta = \Sigma^{-1}\mu,$ Equation (5) becomes

$\begin{eqnarray}\label{geod2} \dot{\Delta} & = & -B\Delta + v\delta^T \ \ {\rm with} \ \ \Delta(0) = I, \delta(0) = 0, \nonumber \\ \dot{\delta} & = & -B\delta + (1+\delta^T\Delta^{-1}\delta)v . \end{eqnarray}$

(6)

Then using

$\begin{equation}\label{A} A = \left( \begin{array}{ccc} -B&v&0 \\ v^T&0&-v^T \\ 0&-v&B \\ \end{array} \right) \end{equation}$

(7)

Eriksen proved that the geodesic solution curve is given by

$\begin{eqnarray}\label{sol} \Lambda:\mathbb{R}:\rightarrow M^k: t\mapsto e^{A t} & = & \left( \begin{array}{ccc} \Delta&\delta&\Phi \\ \delta^T&1+\delta^T\Delta^{-1}\delta&\gamma^T \\ \Phi^T&\gamma&\Gamma \\ \end{array} \right) \end{eqnarray}$

(8)

${\rm where} \ \ \gamma = \Delta^{-1}\delta+\Phi^T\Delta^{-1}\delta, \ {\rm and} \ \ \delta^T\Delta^{-1}\delta~~ = ~~\gamma^T\Gamma^{-1}\gamma.$

(9)

${\rm So}\ \ (\Delta(-t), \delta(-t)) ~~ = ~~(\Gamma(t), \delta(t)).$

(10)

Of course, the analytic difficulty is the requirement to find the length of the geodesic between two points in $M^k$ to obtain a distance function, that being the infimum of arc length over all curves joining the points.

---

2. Approximating distances between arbitrary mixtures of multivariate Gaussians

Here we consider a mixture distribution consisting of a linear combination of $k$-variate Gaussians $\{ f_k, k = 2, 3, \dots , N \}$ with an increasing sequence of $k = 2, 3, \ldots, N$ variables and with probability density functions:

$\begin{equation}\label{kvar} f_2(\mu_2, \Sigma_2), f_3(\mu_3, \Sigma_3)... , f_N(\mu_N, \Sigma_N) \ \ {\rm and} \ \forall k \int_{{\mathbb{R}^k}} f_k = 1 \end{equation}$

(11)

where $\mu_k\in\mathbb{R}^k$ is the $k$-vector of means and $\Sigma_k\in\mathbb{R}^{(k^2+k)/2}$ is the positive definite symmetric $(k\times k)$ covariance matrix with components $(\sigma_{ij}), i\leq j = 1, 2, ..., k.$ The standard basis for the space of covariance matrices is $E_{ij} = 1_{ii} \ {\rm for} \ i = j, \ E_{ij} = 1_{ij}+1_{ji} \ {\rm for} \ i\neq j$ so

$\Sigma = \sum\limits_{i\leq j = 1}^k \sigma_{ij} E_{ij}.$

We presume that the parameters and relative weights $w_k$ of these component probability density functions (11) have been obtained empirically, giving a mixture density:

$\begin{equation}\label{mix} f^A = \sum\limits_{k = 2}^N w_k^A f^A_k, \ \ {\rm with} \ w^A_k\geq 0 \ {\rm and} \ \sum\limits_{k = 2}^N w^A_k = 1. \end{equation}$

(12)

Given two such distributions, $f^A = f^A(\mu^A, \Sigma^A, w^A)$ and $f^B = f^B(\mu^B, \Sigma^B, w^B),$ we wish to be able to estimate the information distance $D(f^A, f^B)$ between them.

There is no general analytic solution for the geodesic distance between two $k$-variate Gaussians, but for many purposes the absolute information distance is not essential and comparative values may suffice for proximity testing, then the sum $D = D_\mu + D_\Sigma$ from (2) and (4) is a natural approximation. Indeed, (4) gives the geodesic distance between $f^A$ with $\Sigma^A = I$ and $f^B$ with $\mu^A = \mu^B = 0$ and the information metric is invariant under affine transformations of the mean ^[3,6].

So, a fortiori, also we do not have the distance between two mixtures of multivariate Gaussians: $f^A(\mu^A, \Sigma^A, w^A)$ and $f^B(\mu^B, \Sigma^B, w^B)$. For this we must resort to approximations for incorporating the weightings of component Gaussians. In practice, it may not matter greatly since the relation between a reasonable approximation and a true geodesic distance is likely to be monotonic, which may be adequate for many applications, and was what we found in our face recognition work ^[12].

---

2.1. Averaging distances over weightings

Perhaps the most natural method is to combine equations (2) and (4) through the linear combination (12), obtaining an approximation as a corresponding linear combination of distances. Given two mixture distributions $f^A(\mu^A, \Sigma^A), f^B(\mu^B, \Sigma^B)$ we could split the distance estimate function $D^\#$ into $D^\#_\mu$ and $D^\#_\Sigma$ as follows with $\delta\mu = (\mu^A-\mu^B)$:

$\begin{eqnarray} D^\#_\mu(f^A, f^B) & = & \sum\limits_{k = 2}^N\frac{1}{2}\left(w_k^A\sqrt{\delta\mu^T\cdot\Sigma_k^{A^{-1}} \cdot \delta\mu} + w_k^B\sqrt{\delta\mu^T\cdot\Sigma^B_k{^{-1}} \cdot \delta\mu } \right) \label{DdmuAv} \end{eqnarray}$

(13)

$\begin{eqnarray} D^\#_\Sigma(f^A, f^B) & = & \sum\limits_{k = 2}^N\frac{1}{2} (w_k^A \sqrt{\frac{1}{2} \sum\limits_{k = 2}^N (\log\lambda _k^{AB})^2} + w_k^B \sqrt{\frac{1}{2} \sum\limits_{k = 2}^N (\log\lambda _k^{BA})^2 }) \label{DSigmaAvg} \ \\ \{\lambda _k^{AB}\} = {\rm Eig}H^{AB}_k , \ \ \ H^{AB}_k & = & \left((\Sigma^A_k)^{-1/2} \cdot \Sigma_k^B \cdot (\Sigma_k^A)^{-1/2} \right).\nonumber \end{eqnarray}$

(14)

We note $D^\#_\mu(f^A, f^B)$ in (13) does give a potentially useful distance measure between the two Gaussian mixtures, since it incorporates both means and covariances. Figure 1 shows the effect on $D^\#_\mu$ values between differing averaged weighting sequences for random $k$-variate Gaussians having $k = 2, 3, 4, 5$ variables, with increasing weights $f^A,$ uniform weights $f^B,$ and decreasing weights $f^C.$ The $g^A, g^B, g^C$ are for the same mixtures except that $\Sigma_2^C$ has been replaced by $\Sigma_2^C/5$ and $h^A, h^B, h^C$ are for the same mixtures except that $\Sigma_5^C$ has been replaced by $\Sigma_5^C/5$ to show the effect of a change in one covariance component. For these simulations Mathematica was used to generate random mean vectors $\mu_i\in[5, 10]$ and random covariance matrix elements $\sigma_{ij}\in[5, 10], i\neq j,$ $\sigma_{ii}\in[-6, 6]$, choices that ensured positive definite symmetric matrices for covariances. This approach was chosen to try to illustrate the effects of weighting sequences and isolated covariance changes on the measurements of distances between mixtures. Table 1 shows mean values $\overline{D^\#_\mu}$ and $\overline{D^\#_\Sigma}$ for the pairs of mixtures $(f^A, f^B), (f^B, f^C), (f^A, f^C)$ for the 20 random Gaussians with weights $w_k^A = (0.1, 0.2, 0.3, 0.4), \ w_k^B = (0.25, 0.25, 0.25, 0.25), \ w_k^C = (0.4, 0.3, 0.2, 0.1)$.

As expected from the form of $D^\#_\Sigma$ in equation (14), its values in the table of means is unaffected by the scale changes in covariances through mixtures $f, g, h,$ but is sensitive to weighting sequences.

However, the $k$-variate components from two mixtures might not come from the same feature space in some applications so there may be no connection between the contributing features they are representing. On the other hand, a commonly used feature space is that of pixel colours in different locations, as for example in texture and face recognition and those feature spaces are the same.

---

2.2. Mixtures projected onto the complex plane

The new implementation described here uses the information geometric norm on the mean vectors and the Frobenius norm on the covariance matrices to project the mixture distributions onto the complex plane. This 2-dimensional representation reveals influences of the means and covariances in the mixtures, which itself may be a valuable It allows also the direct calculation of a distance between two mixture distributions using moduli, without assuming any connections between the mixtures, though this has the effect of smoothing the component influences of the means and covariances.

The idea here is simple: for each mixture distribution $f^A$ given by a weighted sum (12) we obtain two numbers $||\mu^A||$ and $||\Sigma^A||$ being the weighted sums of norms of means and covariances. The norm on mean vectors is given by (3) and for the covariance matrices we need a matrix norm, which here we choose as the Frobenius norm given for an $n\times n$ matrix $M_{\alpha\beta}$ by the square root of the sum of squares of its elements $m_{\alpha\beta},$

$||M_{\alpha\beta}||^2 = \sum\limits_{\alpha = 1}^n\sum\limits_{\beta = 1}^n (m_{\alpha\beta})^2$

Note that if $M_{\alpha\beta}$ has eigenvalues $\{\lambda_\alpha\}$ and is represented on a basis of eigenvectors then

$||M_{\alpha\beta}||^2 = \sum\limits_{\alpha = 1}^n(\lambda_\alpha)^2.$

Given a mixture distribution $f^A$ consisting of $M$ different multivariate Gaussians:

$G^A = \{G^A_i(\mu^A_i, \Sigma^A_i)\}_{i = 1, M}$ with weights $w^A = \{w^A_i\}_{i = 1, M}$ we have

$\begin{eqnarray} f^A & = &\sum\limits_{m = 1}^M w^A_mG^A_m \nonumber \end{eqnarray}$

$\begin{eqnarray} ||\mu^A|| & = & \sqrt{ \sum\limits_{m = 1}^M w^A_m(\mu^A_m)^T.(\Sigma^A_m)^{-1} . (\mu^A_m) } \label{munorm} \end{eqnarray}$

(15)

$\begin{eqnarray} ||\Sigma^A|| & = & \sqrt{ \sum\limits_{m = 1}^M w^A_m||\Sigma^A_m||^2} . \label{Sigmanorm} \end{eqnarray}$

(16)

Now we can represent $f^A$ by the complex number $\phi^A = ||\mu^A||+i ||\Sigma^A||$ and its difference from another such complex number $\phi^B$ for $f^B$ gives us a distance measure in our reduced space of mixtures:

$\begin{equation}\label{deltaf} \Delta(f^A, f^B) = |\phi^B -\phi^A|. \end{equation}$

(17)

Figure 2 shows a plot of the points $(||\mu||, ||\Sigma||)\in \mathbb{C}$ for the 250 mixtures of random $k$-variate Gaussians having $k = 2, 3, 4, 5$ variables, with increasing weights $f^A,$ uniform weights $f^B,$ and decreasing weights $f^C.$ The $g^A, g^B, g^C$ are for the same mixtures except that $\Sigma_2^C$ has been replaced by $\Sigma_2^C/5$ and $h^A, h^B, h^C$ are for the same mixtures except that $\Sigma_5^C$ has been replaced by $\Sigma_5^C/5$ to show the effect of a change in one covariance component. In each case the mean for each over the 250 replications is shown as a large point. This was done using Mathematica with random mean vectors $\mu_i\in[5, 10]$ and random covariance matrix elements $\sigma_{ij}\in[5, 10], i\neq j,$ $\sigma_{ii}\in[-6, 6]$.

---

3. Discussion

There are efficient software programs for extracting from large data sets and image sequences certain mixtures of probability distributions, such as multivariate Gaussians, to represent the important features and their mutual correlations needed for accurate document retrieval from databases. The lack of an analytic solution to the geodesic distance equations between points in the Riemannian space of multivariate Gaussian mixtures, Equation (12), with an information metric, means that approximate solutions need to be found for practical applications. We have illustrated a new approximation for the case of 250 mixtures of $k$-variate Gaussians for $k = 2, 3, 4, 5$ with four weightings $w_k^A = (0.1, 0.2, 0.3, 0.4), \ w_k^B = (0.25, 0.25, 0.25, 0.25), \ w_k^C = (0.4, 0.3, 0.2, 0.1)$ of the component Gaussians that are increasing, uniform and decreasing. These simulations show the effects of covariance changes and the effects of weighting sequences on each given collection of $k$-variate Gaussians for $k = 2, 3, 4, 5.$

---

4. Conclusions

Whereas there are not analytic expressions for the information geometric distance between pairs of mixtures of multivariate Gaussians, we have shown that there are several choices for good information geometric approximate distances which are easy to compute. The new method yielded evident discrimination between pairs of these mixtures, shown in easily interpretable graphical form, Figure 2, distinguishing effects of covariance changes and effects of weighting sequences.

---

Acknowledgement

The methods described in §2.1 were developed with J. Scharcanski and J. Soldera during a visit to UFRGS, Brazil with a grant from The London Mathematical Society in 2013 and the author is grateful to CAPES (Coordeanao de Aperfeioamento de Pessoal de Nivel Superior, Brazil) for partially funding this project. An application of the results to face recognition was reported elsewhere in a joint paper ^[12].

---

Conflict of interest

The author declares that there are no conflicts of interest in this paper.

---