FAST TRAINING ALGORITHMS FOR MULTI-LAYER NEURAL NETS

RICHARD P. BRENT

Abstract

Training a multilayer neural net by back-propagation is slow and requires arbitrary choices regarding the number of hidden units and layers. This paper describes an algorithm which is much faster than back-propagation and for which it is not necessary to specify the number of hidden units in advance. The relationship with other fast pattern recognition algorithms, such as algorithms based on k-d trees, is mentioned. The algorithm has been implemented and tested on artificial problems such as the parity problem and on real problems arising in speech recognition. Experimental results, including training times and recognition accuracy, are given. Generally, the algorithm achieves accuracy as good as or better than nets trained using back-propagation, and the training process is much faster than back-propagation. Accuracy is comparable to that for the “nearest neighbour” algorithm, which is slower and requires more storage space.

Comments

Only the Abstract is given here. The full paper appeared as [1] and was presented as a Forsythe memorial lecture, Department of Computer Science, Stanford University, February 1990. The paper demonstrated a close connection between neural nets and older classification and data retrieval methods in common use by statisticians and computer scientists, and helped to introduce some realism into the debate on the capabilities of neural nets [2].

References

1. R. P. Brent, “Fast training algorithms for multi-layer neural nets”, IEEE Transactions on Neural Networks, 2 (1991), 346–354. Preliminary versions appeared as Report TR-CS-90-01, Computer Sciences Laboratory, ANU, January 1990, 13 pp; and as Technical Report NA-90-03, Department of Computer Science, Stanford University (1990 Forsythe Lecture #2), 16 pp. Extended abstract in Proc. First Australian Conference on Neural Networks (edited by M. Jabri), Electrical Engineering, University of Sydney, January 1990, 97–98. rpb119.

2. F. Crick, “The recent excitement about neural networks”, Nature 337 (1989), 129–132.

Computer Sciences Laboratory, Australian National University, Canberra E-mail address: rpb@cslab.anu.edu.au

rpb119a typeset using AMS-L^{A}TEX

1991 Mathematics Subject Classification. Primary 68T05; Secondary 68Q10, 82C32, 92B20, 92J40.

Key words and phrases. Neural net, training, back-propagation, decision tree, k-d tree, parity problem speech recognition.

Copyright \begin{matrix}c \\ ◯ \end{matrix} 1991, IEEE.

Comments \begin{matrix}c \\ ◯ \end{matrix} 1993, R. P. Brent.