The likelihood

A very popular approach for estimating the ICA model is maximum likelihood estimation, which is closely connected to the infomax principle. Here we discuss this approach, and show that it is essentially equivalent to minimization of mutual information.

It is possible to formulate directly the likelihood in the noise-free ICA model, which was done in [38], and then estimate the model by a maximum likelihood method. Denoting by ${\bf W}=({\bf w}_1,...,{\bf w}_n)^T$ the matrix ${\bf A}^{-1}$, the log-likelihood takes the form [38]:

$$L=\sum_{t=1}^T \sum_{i=1}^n \log f_i({\bf w}_i^T {\bf x}(t))+T\log\vert\det {\bf W}\vert$$ (27)

where the f_i are the density functions of the s_i (here assumed to be known), and the ${\bf x}(t),t=1,...,T$ are the realizations of ${\bf x}$. The term $\log\vert\det {\bf W}\vert$ in the likelihood comes from the classic rule for (linearly) transforming random variables and their densities [36]: In general, for any random vector ${\bf x}$ with density p_x and for any matrix ${\bf W}$, the density of ${\bf y}={\bf W}{\bf x}$ is given by $p_x({\bf W}{\bf x})\vert\det{\bf W}\vert$.