Physical Society Colloquium

How to learn a generative model of images

Geoffrey Hinton

CIAR Neural Computation and Adaptive Perception Program
University of Toronto

I will start by showing how the concept of free energy can be used to justify a simple learning algorithm that is remarkably good at learning to extract features from images of poorly written digits. Then I will show how this simple algorithm can be applied recursively to learn a model of hand-written digits that contains many layers of features. Using the fact that non-equilibrium free energy is always higher than equilibrium free energy, it can be shown that every time we add an extra layer of features the model gets better.

After learning, the top two layers of the model form a high-dimensional Hopfield net which has long, narrow free energy ravines that correspond to the digit classes. The directions along the floor of each ravine represent the allowable variations within that digit class. The patterns of activity in the Hopfield net look nothing like images, but they can be mapped to images via the intermediate layers of features, so it is possible to see what the Hopfield net has in mind. A demonstration of the model is at http://www.cs.toronto.edu/~hinton/digits.html