Redundancy reduction

According to Barlow [6,7,8,9,10] and several other authors [39,4,44,128], an important characteristic of sensory processing in the brain is 'redundancy reduction'. One aspect of redundancy reduction is that the input data is represented using components (features) that are as independent from each other as possible. Such a representation seems to be very useful for later processing stages. Theoretically, the values of the components are given by the activities of the neurons, and ${\bf x}$ is represented as a sum of the weight vectors of the neurons, weighted by their activations. This leads to a linear encoding like the other methods in this Section.

One method for performing redundancy reduction is sparse coding [7,9,44]. Here the idea is to represent the data ${\bf x}$ using a set of neurons so that only a small number of neurons is activated at the same time. Equivalently, this means that a given neuron is activated only rarely. If the data has certain statistical properties (it is 'sparse'), this kind of coding leads to approximate redundancy reduction [44]. A second method for redundancy reduction is predictability minimization [128]. This is based on the observation that if two random variables are independent, they provide no information that could be used to predict one variable using the other one.