Self-Proposed
Supervisor: Chris Williams, 651 1212, ckiw@dai
Proposer: Felix Agakov
Other Suggested Supervisors:
Subject Areas: Machine Learning/Neural Networks/Connectionist Computing,
Project Suitable for the Following Degrees: MSc in AI,
Principal goal of the project: The main aim is to investigate the products of experts (PoE) model when each expert is a Gaussian. We will investigate the maximum likelihood solutions if this model, and analyse the PoE learning rule using results for Gibbs samplers on Gaussians. If there is time, investigations of other PoE architectures can be pursued.
Description:
Recently Geoff Hinton has introduced the Products of Experts architecture [1,2]. The first goal of the project will be to study PoE architectures when each expert distribution is a Gaussian. If the Gaussians are chosen to be Probabilistic Principal Components Analysis models (cf Tipping and Bishop, 1998) with one latent variable then it is expected that the maximum likelihood solution when fitting the product of m models to data will be that they fit the first m principal components.
The second topic is similar to the first, but we consider "pancake" Gaussians where all directions are elongated except one. This is expected to give rise to a minor components analysis solution.
The third topic is to study the nature of the PoE learning rule (one-step Gibbs sampling) in the Gaussian model. There are analytical results concerning Gibbs sampling in Gaussian distributions [3] and the aim is to learn more about the validity of the one-step rule compared to the case when the negative phase of learning is a sample from the joint distribution.
If there is further time, investigations of other PoE architectures can be pursued.
Resources Required: Matlab. Degree of Difficulty: Quite hard, needs good mathematical skills.
References: