Encyclopædia BayesICA

"The physics of learning systems"
Probability is a measure of degree of belief
Bayes rule tells how beliefs are updated when new observations are made
Marginalization can be used for prediction and inference about unknown quantities
Decisions are based on expected utilities (where expectation is taken over the current beliefs/probabilities)
In practice, the integrals are approximated in some way or another

Bayesian ensemble learning is one type of variational learning
Misfit between the true posterior and its approximation is measured by Kullback-Leibler information
Yields bounds for probability of the data given the model structure
Close connection to minimum description length principle

Calculus of variations refers to optimization problems where the optimand is a function (as opposed to real number or a vector of real numbers)
Methods where a simpler function is fitted to the posterior

Models which postulate unobserved variables to explain regularities in the observations
Gaussian linear factor analysis model is one of the simples examples: unknown factors are assumed to be responsible for the observations
Pretty much the same as generative models

Models which explicitly state how the observations have been generated by partly or fully unknown factors
Pretty much the same as latent variable models

The goal of unsupervised learning is to extract an efficient representation of the statistical structure implicit in the observations
Generative or latent variable models are common tools

Principle which states that the most compact explanation for the observations should be favoured over other explanations
Description length L(x) of x is measured in bits and is (more or less) L(x) = -log P(x), where P(x) is the probability of x
As a learning theory not as well founded theoretically as Bayesian statistics but can allow intuitive interpretations to various phenomena in learning
Bayesian ensemble learning bridges Bayesian statistics and MDL