Encyclopædia BayesICA

Bayesian statistics

• "The physics of learning systems"
• Probability is a measure of degree of belief
• Bayes rule tells how beliefs are updated when new observations are made
• Marginalization can be used for prediction and inference about unknown quantities
• Decisions are based on expected utilities (where expectation is taken over the current beliefs/probabilities)
• In practice, the integrals are approximated in some way or another

Ensemble learning

• Bayesian ensemble learning is one type of variational learning
• Misfit between the true posterior and its approximation is measured by Kullback-Leibler information
• Yields bounds for probability of the data given the model structure
• Close connection to minimum description length principle

Variational learning

• Calculus of variations refers to optimization problems where the optimand is a function (as opposed to real number or a vector of real numbers)
• Methods where a simpler function is fitted to the posterior

Latent variable models

• Models which postulate unobserved variables to explain regularities in the observations
• Gaussian linear factor analysis model is one of the simples examples: unknown factors are assumed to be responsible for the observations
• Pretty much the same as generative models

Generative models

• Models which explicitly state how the observations have been generated by partly or fully unknown factors
• Pretty much the same as latent variable models

Unsupervised learning

• The goal of unsupervised learning is to extract an efficient representation of the statistical structure implicit in the observations
• Generative or latent variable models are common tools

Minimum description length (MDL) principle

• Principle which states that the most compact explanation for the observations should be favoured over other explanations
• Description length L(x) of x is measured in bits and is (more or less) L(x) = -log P(x), where P(x) is the probability of x
• As a learning theory not as well founded theoretically as Bayesian statistics but can allow intuitive interpretations to various phenomena in learning
• Bayesian ensemble learning bridges Bayesian statistics and MDL

Figure caption

Pine trees in winter in Espoo.