ALGORITHMS FOR MATRIX MULTIPLICATION

R. P. BRENT

Abstract

Strassen’s and Winograd’s algorithms for n × n matrix multiplication are investigated and compared with the normal algorithm.

The normal algorithm requires n^{3} + O(n^{2}) multiplications and about the same number of additions. Winograd’s algorithm almost halves the number of multiplications at the expense of more additions. Strassen’s algorithm reduces the total number of operations to O(n^{2.82}) by recursively multiplying 2n × 2n matrices using seven n × n matrix multiplications.

Floating-point error bounds are obtained for both Strassen’s and Winograd’s methods. It is shown that Strassen’s method satisfies a certain numerical stability property (albeit weaker than that satisfied by the normal method); and that scaling is essential for numerical accuracy using Winograd’s method.

In practical cases, for moderate n, Winograd’s method appears to be slightly faster than the other two methods, but the gain is, at most, about 20 percent. Strassen’s method should be faster for sufficiently large n, and this will be important in the future as memory sizes increase.

An attempt to generalize Strassen’s method is described.

Comments

This report describes a project undertaken as one of the requirements for the degree of Master of Science, and hence may be regarded as the author’s master’s thesis. Only the Abstract is given here. The full report appeared as [1]. For related work, see [2, 3, 4].

The error analysis of Strassen’s algorithm shows that, although it is not as stable componentwise as the normal algorithm, it is sufficiently stable to be usable in implementations of Level 3 Basic Linear Algebra Subroutines (BLAS). (This was not pointed out in the report since Level 3 BLAS had not been invented at the time.)

rpb002a typeset using AMS-L^{A}TEX.

1991 Mathematics Subject Classification. Primary 68Q25; Secondary 65F05, 65F35, 65F30, 65G99.

Key words and phrases. Matrix multiplication, inner-product, error analysis, scaling, Winograd, Strassen, computational complexity.

CR Categories. 5.11, 5.14, 5.19, 5.39.

Reproduction in whole or in part is permitted for any purpose of the United States Government.

The preparation of this manuscript was supported in part by the Office of Naval Research (NR 044 211), the National Science Foundation (GJ 798), and the Atomic Energy Commission (Stanford PA #18).

The author thanks R. Floyd and J. Herriot for their helpful advice, and CSIRO (Australia) for its generous financial support.

Copyright \begin{matrix}c \\ ◯ \end{matrix} 1970, R. P. Brent.

Comments \begin{matrix}c \\ ◯ \end{matrix} 1993, 2000, R. P. Brent.

References

[1] R. P. Brent, Algorithms for Matrix Multiplication, Report TR-CS-70-157, Computer Science Department, Stanford University (March 1970), 52 pp. Available from NTIS, #AD705509. Page im-

   ages available from ftp://elib.stanford.edu/pub/reports/cs/tr/70/157/GIF-PAGE-IMAGES/ and from
   ftp://ftp.comlab.ox.ac.uk/pub/Documents/techpapers/Richard.Brent/rpb002/

[2] R. P. Brent, “Error analysis of algorithms for matrix multiplication and triangular decomposition using Winograd’s identity”, Numerische Mathematik 16 (1970), 145–156. MR 43#5702, CR 12#21408. rpb004.

[3] V. Strassen, “Gaussian elimination is not optimal”, Numerische Mathematik 13 (1969), 354–356.

[4] S. Winograd, “A new algorithm for inner product”, IEEE Transactions on Computers C-17 (1968), 693–694.

Department of Computer Science, Stanford University, Stanford, CA 94305, USA