Fast training algorithms for multi-layer neural nets

119. R. P. Brent, Fast training algorithms for multi-layer neural nets, IEEE Transactions on Neural Networks 2 (1991), 346-354.

Abstract: dvi (3K), pdf (29K), ps (26K).

Paper: dvi (31K), pdf (153K), ps (83K).


Training a multilayer neural net by back-propagation is slow and requires arbitrary choices regarding the number of hidden units and layers. This paper describes an algorithm which is much faster than back-propagation and for which it is not necessary to specify the number of hidden units in advance. The relationship with other fast pattern recognition algorithms, such as algorithms based on k-d trees, is mentioned.

The algorithm has been implemented and tested on artificial problems such as the parity problem and on real problems arising in speech recognition. Experimental results, including training times and recognition accuracy, are given. Generally, the algorithm achieves accuracy as good as or better than nets trained using back-propagation, and the training process is much faster than back-propagation. Accuracy is comparable to that for the "nearest neighbour" algorithm, which is slower and requires more storage space.


The paper was based on a George and Sandra Forsythe memorial lecture, Department of Computer Science, Stanford University, February 1990. It demonstrated a close connection between neural nets and older classification and data retrieval methods in common use by statisticians and computer scientists, and helped to introduce some realism into the then "overheated" debate on the capabilities of neural nets [see F. Crick, "The recent excitement about neural networks", Nature 337 (1989), 129-132].

For a paper dedicated to George and Sandra Forsythe, see [238].

Go to next publication

Return to Richard Brent's index page