The Systems Performance Evaluation Co-operative was set up at the end of last year with the intention of developing a sensible benchmarking suite for comparing the performance of various workstation technologies. It claims that existing measures like MIPS are often manipulated by manfacturers and misinterpreted by the press and users. The Co-op has now come […]
The Systems Performance Evaluation Co-operative was set up at the end of last year with the intention of developing a sensible benchmarking suite for comparing the performance of various workstation technologies. It claims that existing measures like MIPS are often manipulated by manfacturers and misinterpreted by the press and users. The Co-op has now come up with its first set of bechmarks, and the all important results of testing them on various architectures, which seems to demonstrate the superiority of RISC over complex instruction set working. The initial Release 1.0 set consists of a Unix-based suite of 10 Fortran and C benchmarks that examine the performance of both integer and floating point computation in scientific and technical applications. These are available on a tape from the Co-op for $450, and SPEC also is also publishing a quarterly newsletter containing benchmark results, which costs $150. Future suites will measure other aspects of system performance, such as disk, graphics and communications, and will measure system performance in networked and multi-user environements, as well as applications. A Spec Reference Time, which works out at 3867.7 for Release 1.0, is the average time in seconds that it takes a DEC VAX-11/780 machine to run the complete set of tests. A Spec Ratio is derived by dividing the Spec Reference Time by the time each machine takes to run the 10 benchmarks, and the single figure that summarises the 10 tests is the Specmark, which is the geometric mean of the Spec Ratio result for each of the 10 benchmarks. The table gives the overall results of the tests.
Machine Processor Clock Time Specmark MIPS M/2000 R3000 25MHz 235.0 16.5 Motorola Delta 8864DP 2x 88100 20MHz 511.8 15.1 Apollo DN10010 Prism 18.2MHz 267.0 14.5 Sun Sparcstation 330 CYC7C601 25MHz 343.7 11.3 MIPS M/120-5 R2000 16.67MHz 345.3 11.2 DECstation 3100 R2000 16.67MHz 381.4 10.1 HP9000 Model 834 PA-RISC 15MHz 408.0 9.5 MIPS RC2030 R2000 16.67MHz 417.6 9.3 Sun Sparcstation 1 MB86901 20MHz 468.5 8.3 Motorola Delta 8864SP 88100 20MHz 473.3 8.2 Motorola Delta 8608 88100 20MHz 496.5 7.8 DECstation 2100 R2000 12.5MHz 518.7 7.5 HP9000 Model 370 68030 33MHz 980.3 3.9 HP9000 Model 340 68030 16.67MHz 2432.4 1.6
The MIPS Computer M/2000 emerges as a clear winner in the performance stakes. Not surprisingly the Motorola Delta 8864DP machine comes in at second place – it has two processors. What is surprising however is DEC’s standing – down in sixth place since its DECstations use the same MIPS chip, as MIPS’ own M/120 5 system, yet perform less well, suggesting that DEC’s implementation has lost something in translation – having been brought up from its infancy in the world of proprietary architectures, DEC finds it difficult to shake the habit even while paying lip service to the brave new world of open systems and machine-independent industry standards, and it has introduced one little subtle incompatibility into its MIPS implementation, so that applications written for other machines based on the MIPS RISC have to be recompiled for the DECstations – and word is that the version of the next generation MIPS chip that DEC plans to fabricate will be substantially more incompatible. It is noteworthy that for example, a 25MHz Sparc performs much less well than a 25MHz R3000 from MIPS, and that the fastest chip in the list is the 33MHz Motorola 68030 – yet it is outperformed by all the RISCs. Apollo Computer Inc, Hewlett-Packard Co, MIPS Computer Systems, Sun Microsystems and Electronic Engineering Times were the original founders – they have since been joined by AT&T Co, Control Data Corp, DEC, Data General, IBM, Intergraph Corp, Motorola Inc, Multiflow Computers Inc and Stardent Inc.