** Next:** Blind deconvolution
** Up:** Higher-order methods
** Previous:** Projection pursuit

###

Redundancy reduction

According to Barlow [6,7,8,9,10] and
several other authors [39,4,44,128],
an important characteristic of sensory processing in the brain is
'redundancy reduction'. One aspect of redundancy reduction is that the
input data is represented using
components (features) that are as independent from each other as possible.
Such a representation seems to be very useful for later processing stages.
Theoretically, the values of the components are given by the activities of the
neurons, and
is represented as a sum of the
weight vectors of the neurons, weighted by their activations. This
leads to a linear encoding like the other methods in this Section.

One method for performing redundancy reduction is sparse coding
[7,9,44]. Here
the idea is to represent the data
using a set of neurons
so that only a small number of neurons is activated at the same time.
Equivalently, this means that a given neuron is activated only rarely.
If the data has certain statistical properties (it is 'sparse'), this
kind of coding leads to approximate redundancy reduction [44].
A second method for redundancy reduction is predictability
minimization [128]. This is based on the observation that
if two random variables are independent, they provide no information
that could be used to predict one variable using the other one.

** Next:** Blind deconvolution
** Up:** Higher-order methods
** Previous:** Projection pursuit
*Aapo Hyvarinen*

*1999-04-23*