
R 1.9.0, the latest version of our reference software, a rich and powerful free 'S language dialect' (R benchmark 2.3 script; text file, 13 Kb). Here, we use the Pentium IVoptimized ATLAS library (provided on CRAN), which gives slightly better results in some tests. We have not tested other optimized libraries, like Goto's one.  
SPLUS 6.1, the commercial equivalent of R (SPLUS benchmark 2 script; text file, 10 Kb)  
Matlab 6.0 (R12), our previous reference (download Matlab benchmark 2 script and accompanying gcd2.m custom function; text file, 10 Kb). Warning! This is not the latest version. At the time I write this (21 April  
OMatrix 5.6, a cheap but very fast package, that can run most Matlab scripts (OMatrix native mode benchmark 2 & OMatrix Matlab mode benchmark 2 scripts; text files, 10 Kb each)  
Octave 2.1.42, a free "clone" of Matlab 4 (Octave benchmark 2 script; text file, 10 Kb). The version used was compiled with an optimized ATLAS library for the Pentium IV.  
Scilab 2.7, a very complete free software, "not unlike" Matlab (Scilab benchmark 2 script; text file, 10 Kb)  
Ox 3.30, a very efficient matrix package similar to Gauss and free for academic use (Ox benchmark 2 script; text file, 11 Kb) 
Tests are:
I. Matrix calculation: evaluates the ability of performing some common matrix computations.
I.A: creation, transposition, deformation of a 1500x1500 matrix. This test evaluates the ability to create and manipulate matrices.  
I.B: creation of a 800x800 normally distributed random matrix and taking the 1000^{th} power of all its elements. Evaluates the speed at which a random matrix is processed element by element.  
I.C: sorting of 2,000,000 random values. Tests the speed of a sorting operation.  
I.D: 700x700 crossproduct matrix (b = a' * a). Evaluates matrix operations.  
I.E: linear regression over a 600x600 matrix (b = a \ b'). Tests the speed of execution for linear models evaluation. 
II. Matrix functions: evaluates speed of some preprogrammed matrix functions.
II.A: fast Fourier transform over 800,000 values. Fourier transform is a commonly used method in signal processing.  
II.B: eigenvalues of a 320x320 random matrix. Eigenvalues are used in multivariate analyses (PCA, ...).  
II.C: determinant of a 650x650 random matrix. Calculation of the determinant of a matrix is a common, but unequally optimized, function in matrix calculation packages.  
II.D: cholesky decomposition of a 900x900 matrix. Another commonly preprogrammed function.  
II.E: inverse of a 400x400 random matrix. A computationally intensive function for which various algorithms exist (with very different performances). 
III. Programming: evaluates efficiency to run scripts and custom functions.
III.A: 750,000 Fibonacci numbers calculation. This evaluates the speed of vector calculation.  
III.B: creation of a 2250x2250 Hilbert Matrix. Evaluates performances in matrix calculation in scripts.  
III.C: grand common divisors of 70,000 pairs. Tests potentials in using recursive functions.  
III.D: creation of a 220x220 Toeplitz matrix. Check the speed of execution for loops.  
III.E: Escoufier's method on a 37x37 random matrix. Tests various aspects of programming combined in a single test. 
Note that tests III.AE are not most optimized algorithms for each package, but they do test similar features in all of them. For instance, a matrix algorithm for test III.D is often much more efficient, as is a possibly preprogrammed toeplitz() function. Yet, we keep the loop algorithm in all cases... in order to test the speed of loops execution in scripts!
The tests were run three times on a Pentium IV 1.6 Ghz computer with 1 Gb of memory under Windows XP professional and the mean value is recorded. The next table presents results:
Test (sec)  R 1.9.0  SPLUS 6.1  Matlab 6.0  OMatrix 5.6 Ml mode 
OMatrix 5.6 native  Octave 2.1.42  Scilab 2.7  Ox 3.30 
I. Matrix calculation  
I.A  1.49  3.03  0.48  0.69  0.58  2.01  1.19  0.74 
I.B  0.43  1.37  0.42  0.53  0.62  1.22  0.70  0.94 
I.C  0.87  2.38  0.89  0.98  0.98  7.77  2.00  1.97 
I.D  0.26  0.72  0.73  0.19  0.30  0.35  8.58  0.45 
I.E  0.26  1.33  0.24  0.17  0.14  0.78  2.11  1.04 
Score  0.46  1.63  0.53  0.41  0.48  1.24  1.71  0.90 
II. Matrix functions  
II.A  1.01  1.62  0.48  0.99  1.05  0.96  1.78  3.06 
II.B  1.25  0.96  0.86  0.41  0.49  2.30  2.44  1.78 
II.C  0.30  0.41  0.27  0.13  0.14  1.02  2.27  0.71 
II.D  0.24  1.92  0.33  0.11  0.12  0.21  1.96  0.36 
II.E  0.14  1.48  0.23  0.07  0.06  0.47  1.67  0.35 
Score  0.42  1.35  0.35  0.18  0.20  0.77  2.00  0.77 
III. Programming  
III.A  0.83  1.68  2.11  0.31  1.84  2.06  0.72  0.69 
III.B  1.33  1.14  0.84  0.51  0.64  0.73  0.91  0.79 
III.C  0.56  0.71  0.91  0.14  0.17  0.42  1.52  0.72 
III.D  0.67  6.62  0.38  0.10  0.10  4.39  1.45  0.05 
III.E  0.89  15.10  1.92  0.60  0.56  3.08  3.97  0.31 
Score  0.79  3.15  1.14  0.28  0.39  1.67  1.26  0.54 
Total  10.52  40.47  11.12  5.93  7.83  27.76  33.27  13.97 
Overall  0.53  1.71  0.60  0.27  0.34  1.17  1.63  0.72 
The higher the result (in seconds), the slower the test executes. Low values mean thus higher performances. Results lower than 0.50 (more than twice faster than the reference) are in green; result larger than 2.00 (more than twice slower than the reference) are in violet. We immediately see the progress made in R since version 1.6.2 (about 30% faster, but as much as four to seven times faster for some operations using the optimized libraries).
SPLUS is a wellrecognized standard in statistics, and it is the commercial counterpart of R. As we see here, it is much more slower than R under Windows (it takes four times more to complete all tests)! SPLUS is wellknow for its versatility, and for the ease of exploring statistical models in its environment. It excels in almost all fields of statistics. However, its limits are reached when working with huge datasets. In this case, SAS (not evaluated here) is considered to be faster, and thus more efficient, especially in loops programming where SPLUS is desperately slow (test III.E)! However, SPLUS propose alternatives: the For() function for optimized loops, and the apply() family of functions that "vectorize" loops. With middlesize matrices, as in the current test, it is easily outperformed by almost all the other software evaluated here.
Since R offers similar features than SPLUS, a larger number of additional libraries (more than 300!), and is totally free, it is clearly an excellent choice for statistical analyses. This benchmark shows also that it is also quite good for "number crunching". Moreover, it runs on almost all platforms (Windows, Macintosh, Unix/Linux) and it has not the "loop problem" of SPLUS (yet it also provides apply() and the like to accelerate loops). However, it does not propose (yet) the same nice user interface with menus and dialog boxes (GUI) as SPLUS 6.1 does,... (though many professionals do not care about that because they prefer to use scripts and the command line for a finer control on their calculations). R becomes better and better with the successive releases. It is maintained and enriched by a very active community of developers. These are the reasons why we decided to promote it as a reference in our benchmark tests.
Matlab 6 is a commercial standard in pure matrix calculation. It is significantly poorer in statistical models than SPLUS or R, but it offers a wide range of highquality toolboxes for specific applications (although, they increase the cost of this already very expensive software!). Concerning speed, it is about as fast as R 1.9.0. However, we did not tested the latest version, 6.5.1, that seems to provide some substantial increase in speed. As being one of the fastest, the richest, the most commonly used and having one of the best user interface, Matlab 6 deserves its status of leading product in matrix programming.
Matlab has several contenders that propose a similar matrix language for a lower price (OMatrix, Octave, Scilab). Among them, only one is fighting also on the performance level with Matlab 6.0: OMatrix. Overall, OMatrix is the fastest matrix computation package we have tested. It is much less expensive than Matlab, and it provides reasonable compatibility. However, OMatrix does not propose the same range of specialized toolboxes and it runs only on Windows.
The two other "Matlab clones" (Octave & Scilab) are free open source software. Their performances are somewhat lower than Matlab 6.0 and better compare with Matlab 5.3 (see version 1 of the test). Octave aims to be fully compatible with the base version of Matlab 4.2. One should note that Octave runs under the cygwin emulation of Unix in Windows, and this has probably some negative impact on its pure performances. The Unix/Linux native version should run comparatively faster. Scilab proposes many more functions than Octave, but it is not 100% compatible with the Matlab language, and it is the slowest package of this comparison if we except the "loop problem" of SPLUS.
Ox is a little apart. It is the only package that does not claim compatibility with one of the two standards previously cited: Matlab or SPLUS. However, it is partly compatible with Gauss, another high quality commercial matrix calculation software regarded as a standard in econometry (not evaluated here, but you will find detailed tests in Stephan Steinhaus' report). It is one of the four software (with R 1.9.0, Matlab and OMatrix) to be faster than R 1.6.2, that is, our reference software and version for this benchmark. It is particularly good for the execution of scripts (tests III). As it is a lightweight console application that can easily run scripts in batch mode, Ox is an excellent choice to shell matrix calculation scripts in various kind of applications. OMatrix is even faster, but it is restricted to Windows systems.
The choice of a data analysis software is a difficult task. "Matrix languages" (like all the software we evaluated here) are very flexible because they are programmable and they are able to work very efficiently with matrices (by definition!) that are widely used in data analysis. However, they differ from each other in term of price, richness (the number of function provided), usability (including the quality of their user interface, their status of established standard or not, the quality of their support, their availability on different platforms like Windows, Macintosh, Unix or Linux), and finally, in term of their pure performances. We evaluated the latter here by using a benchmark suite of 15 tests. Considering results obtained with our benchmark (but beware of its limits: only few features were tested, and solely on a Windows platform!), one can conclude:
R is one of the fastest open source data analysis packages. Since it is free and provides many additional
packages for all kind of statistics, we warmly recommend it.SPLUS is slower and much more expensive, but it still offers a better graphical user interface. Matlab is equally fast, rich and offers a welldesigned user interface, but it is equally expensive. OMatrix is the fastest matrix language we have tested on Windows. Currently, no free "clone" of Matlab is as fast as Matlab 6.0 itself. Octave is languagecompatible with Matlab, but not a top performer on Windows. Scilab is a free alternative of Matlab for "richness" more than for performance. Ox is a very efficient matrix language, especially for batch process of scripts.
Last update: 08/03/2003 