Laboratory of Computer and Information Science / Neural Networks Research Centre CIS Lab Helsinki University of Technology

Theoretical improvements

Besides the effect of posterior approximation mentioned already before and studied in (Ilin and Valpola, 2005), our major theoretical results include more accurate linearization for learning nonlinear models, and treatment of partially observed values.

More accurate linearization for nonlinear models

Variational Bayesian learning of nonlinear models fundamentally reduces to evaluating statistics of the data predicted by the model. The model is a function of the parameters of the variational approximation of the posterior distribution. This is equivalent to evaluating statistics of a nonlinear transformation of the approximating probability distribution. In our earlier works on nonlinear models, we used a Taylor series approximation to linearize the nonlinearity. Unfortunately this approximation breaks down when the variance of the approximating distribution increases, and this leads to algorithmic instability.

For handling this problem, a new linearization method based on a more global Gauss-Hermite quadrature approximation was proposed in (Honkela and Valpola, 2005). The new linearization method yields significantly more accurate estimates of the cost of the model while being computationally almost as efficient.

Partially observed values

It is well-known that Bayesian methods provide well-founded and straightforward means for handling missing values in data. The same applies to values that are somewhere between observed and missing. So-called coarse data means that we only know that a data point belongs to a certain subset of all possibilities. So-called soft or fuzzy data generalises this further by giving weights to the possibilities. In (Raiko, 2004), different ways of handling soft data are studied in context of variational Bayesian learning.


A. Honkela and H. Valpola, "Unsupervised variational Bayesian learning of nonlinear models." In L. Saul, Y. Weiss, L. Bottou (Eds.), Advances in Neural Information Processing Systems 17, pp. 593-600, 2005, MIT Press. Pdf (118k)

A. Ilin and H. Valpola, "On the effect of the form of the posterior approximation in variational learning of ICA models." Neural Processing Letters 22(2), 2005, pp. 183-204. Publisher electronic edition

T. Raiko, "Partially observed values". In Proc. of the 2004 IEEE Int. Joint Conf. on Neural Networks (IJCNN2004), Budapest, Hungary, July 2004, pp. 2825-2830. Pdf (119k)

You are at: CISResearchAlgorithmsBayes → Research

Page maintained by juha.karhunen at, last updated Wednesday, 01-Oct-2008 10:28:24 EEST