Tests are:

I. Matrix calculation: evaluates the ability of performing some common matrix computations.

II. Matrix functions: evaluates speed of some preprogrammed matrix functions.

III. Programming: evaluates efficiency to run scripts and custom functions.

Note that tests III.A-E are not most optimized algorithms for each package, but they do test similar features in all of them. For instance, a matrix algorithm for test III.D is often much more efficient, as is a possibly preprogrammed toeplitz() function. Yet, we keep the loop algorithm in all cases... just to test the speed of loops execution in scripts!

Results

The tests were run three times on a Celeron 500 Mhz computer with 256 Mb of memory under Windows 2000 professional and the mean value is recorded. The next table presents results:

Comments

The higher the result (in seconds), the slower the test executes. Low values mean thus higher performances. Results lower than 0.90 (faster than the reference) are in green; result larger than 5.00 (more than five times slower than the reference) are in violet. We immediately see that Matlab 6.0 tends to be faster (as an overall trend) than the others in categories I and II, but slower in category III (programming).

Matlab 6.0 uses more recent, and more optimized matrix calculation libraries than Matlab 5.3. Consequently, Matlab 6.0 is more than twice faster than Matlab 5.3! And with a much better user interface, it is worth the while to upgrade to Matlab 6.x if you still use an older version! On the other hand, scripts are not faster (unless they use optimized preprogrammed functions, of course). Matlab is a well-recognized standard in many fields requiring matrix computation, like signal processing for instance. As being one of the fastest, the richest, the most commonly used and having one of the best user interface, we choose Matlab 6.0 as a reference for matrix programming software. However, it is quite expensive (comparing to the open source alternatives), and often has to be expanded with optional toolboxes!

Splus is another well-recognized standard, this time in statistics, a field where Matlab is comparatively weaker, even using additional toolboxes. Splus is also very expensive, but it excels in almost all fields of statistics. It is appreciated by professionals for its versatility, and for the ease of exploring statistical models in its environment. Its limits are reached when working with huge datasets. In this case, SAS (not evaluated here) is considered to be faster, and thus more efficient. In our tests, Splus is somewhat slower than Matlab 6.0, especially in loops programming where it is desperately slow! However, Splus propose alternatives: the For() function for optimized loops, and the apply() family of functions that "vectorize" loops. If it is slower than Matlab, it is also because many functions calculate extra diagnostics useful for statistical analysis. This is not taken into account in our raw benchmark, but could turn out to be a decisive advantage in practice, depending on the use of the calculation.

R is developed by a very active community of statisticians and is evolving quickly. At the moment, it proposes about the same panel of functions as Splus does... and it is totally free! It also runs on almost all platforms (Windows, Macintosh, Unix/Linux) and it has not the "loop problem" of Splus (and also provides apply() and the like to accelerate loops). However, it does not propose the same nice user interface with menus and dialog boxes (GUI) as Splus 6 does,... (though many professionals do not care about that because they prefer to use scripts and the command line for a finer control on their calculations). It is also somewhat slower than Matlab 6.0, but compares to Matlab 5.3. Some serious lacks in performances (noted with this test in earlier versions) are now corrected, and it behaves more homogeneously in all tests. It is object-oriented, meaning it is a little harder to learn than Matlab, but much more flexible once one understands its object-oriented features. The same remarks as for Splus can be also formulate for R.

It is also possible to recompile R for using an optimized linear calculus library (ATLAS) instead of the standard BLAS library used by default. On the test computer, with ATLAS optimized for winNT, PII with SSE1, gains are mainly visible in test I.D (cross-product), which drops from 1.90 sec to 0.59 sec. In a lesser extend, test II.C (determinant) is also optimized: from 5.82 sec to 4.53 sec). Although significant, these improvements do not change its position in the midrange, just half-way between fastest and slowest software. R is an excellent choice for those who are looking for statistical analysis because it offers similar features as Splus and it is free and faster! It compares with Matlab 5.3 in most tests, both in matrix computation and programming ability.

As being a well-recognized standard, Matlab has several contenders that propose similar features for lower price (O-Matrix, Octave, Scilab, Rlab). Among them, only one is fighting also on the performance level with Matlab 6.0: O-Matrix. Overall, O-Matrix is the fastest matrix computation package we have tested. It is much less expensive than Matlab, and it provides reasonable compatibility. However, O-Matrix does not propose the same range of specialized toolboxes and it runs only on Windows.

The three others (Octave, Scilab & Rlab) are all free open source software. Their performances are somewhat lower than Matlab 6.0 and better compare with Matlab 5.3. Octave aims to be fully compatible with the base version of Matlab 4.2. Its strongest point, among free Matlab "clones" is its language compatibility. It is also the fastest where -presumably- the ATLAS library is used. One should note that Octave runs under the cygwin emulation of Unix in Windows, and this has probably some negative impact on its performances. The Unix/Linux native version should run comparatively faster.

Scilab proposes many more functions than Octave, but it is not 100% compatible with the Matlab language. Rlab is neither fully compatible, nor very rich. Moreover, its development seems to be stopped. However, since its source code is distributed and modifiable, someone else could continue the project (although it is perhaps wiser to contribute to ongoing open source projects: either Octave, or Scilab).

Ox is a little apart. It is the only package that does not claim compatibility with one of the two standards previously cited: Matlab or Splus. However, it is partly compatible with Gauss, another high quality commercial matrix calculation software regarded as a standard in econometry (not evaluated here, but you will find detailed tests in Stephan Steinhaus' report). Its performances for categories I and II are closer (although a little bit better) to Matlab 5.3 than to Matlab 6.0. However, regarding category III, programming, it is more than twice as fast as Matlab 6.0! As it is a lightweight console application that can easily run scripts in batch mode, Ox is an excellent choice to shell matrix calculation scripts in various kind of applications. O-Matrix is even quicker, but it is unfortunately not embeddable in a custom application.

Conclusions