The need for a realistic way of comparing router performance was highlighted this week, with disagreement between Wellfleet Communications Corp and Cisco Systems Inc on the validity of a new set of tests. To date, the battle for supremacy in the network component market has largely centred around raw speed. Suppliers have tended to focus […]
The need for a realistic way of comparing router performance was highlighted this week, with disagreement between Wellfleet Communications Corp and Cisco Systems Inc on the validity of a new set of tests. To date, the battle for supremacy in the network component market has largely centred around raw speed. Suppliers have tended to focus attention on the protocol handling abilities of their products, often boasting of speeds greater than 100,000 frames per second. But as router technology improves, some suppliers argue, such figures become less and less relevant. Manufacturers normally quote performance figures produced under ideal conditions, and products are already beginning to approach the theoretical limit of performance.
These figures bear little relationship to the performance a user could expect to see in real life. The controversy between Cisco and Wellfleet started when the latter put out a statement announcing the results of tests carried out in the US on its Backbone Node bridge-router products. The figures, produced in IP routing tests at Harvard University and New Jersey-based InterLAB, are said by Wellfleet to shatter Cisco’s speed claims for its recently-launched 7000 range. In a 10-stream IP-routing Ethernet test – between five pairs of boxes – Wellfleet’s Backbone Concentrator Node performed at 146,000 packets per second, against 115,000pps for the Cisco 7000. In a two-stream FDDI test, the Wellfleet Backbone Link Node scaled to over 146,000pps IP routing. No comparable figure was given for a Cisco product. A consultant at Harvard University’s Office of Information Technology commented, The Wellfleet Backbone Concentrator Node raised performance to near theoretical limits across the entire protocol suite tested. But Jurgen Obermann, European marketing manager for Cisco, claims that the figures are meaningless in evaluating the true performance of the product in an internetwork, where protocols other than Ethernet would be involved. He believes that the concept of a theoretical limit is part of the problem, and that the tests demonstrate only the capabilities of the hardware in isolation. Put the router into a true internetwork environment, he says, and the picture would be very different. The Harvard University and InterLAB test suite does include a number of different protocols – IP, IPX, Bridge and others. Barry Sercombe, senior systems engineer with Wellfleet, claims that these tests – which give consistently high results for the Wellfleet products – provide a meaningful rating of performance. But, as Obermann points out, these tests take no account of the proportion of traffic that would actually be routed in a normal network.
By Emma Woollacott
With 30 Ethernet connections, even if traffic was exceptionally heavy (say 1,200pps), on a typical network 25% of this traffic might be expected to be passed through the router – or 300pps. Multiplying 300pps by the 30 connections still gives a figure of only 9,000pps – far lower than the packets-per-second rating of either product. The difference between the two products at rates above 100,000pps is therefore unimportant, says Obermann. But this argument highlights the difficulty in testing routers in a real-life scenario. There is no clear way of defining an ‘average’ network carrying ‘average’ traffic, comparable with the applications benchmarks used by computer suppliers, as there are just too many variables. Quite apart from considerations of network architecture and real-life throughput, there’s the question of average packet size. Mark Raymond, a senior analyst with Datapro, points out, When a single network medium is considered in its own right, the speed at which it operates does not necessarily relate to the overall network performance, he said. The reason is that in a large network the system comprises many different components with applications operating over them. If you consider, say, a wide area internet with a client-server application operating across it, the application will generate small packets in a transaction pro
cessing environment and very large packets in a bulk data transfer. This creates a situation where the performance of the application is governed by the overall activity of the network, Token Ring supplier Olicom A/S recently endorsed this view, announcing that it would no longer publish raw protocol handling figures for its products. Instead, it plans to publish performance figures produced by testing in a networked environment, so that the results take account of the software overhead associated with the protocol stacks of network operating systems such as Novell Inc NetWare or Microsoft Corp’s LAN Manager. In addition, the company plans only to use test frames which it claims approximate an average load. InterLAB and Harvard University test throughput for 64-byte, 1,024-byte and 4,477-byte packets.
But a spokesman for Olicom quotes separate tests carried out by InterLAB in a NetWare environment that show that 82% of all frames transmitted are less than 512 bytes in size; Olicom now plans to measure performance in terms of average-sized frames per second. John Elliott, managing director of independent testing lab GCO, agrees that raw protocol handling is a poor measure of component performance, and suggests that using such tests actually hinders the industry. Router manufacturers end up optimising their routers purely so that they get a good rating from [Harvard University’s] Scott Bradner, he says. GCO uses a test suite based around Clipper-compiled dBase, which although it allows users to specify record and file sizes prevents the tester from making any decisions about block sizes for reads and writes or network packet sizes. The idea is to test network components under the same conditions as would be experienced by a normal network user. The problem is unlikely to be resolved. Even if test labs and suppliers were to be able to agree on a standard environment and test suite, canny manufacturers would optimise their products for good performance in those tests, presumably at the expense of other abilities. The ideal method would be to test products in absolutely every conceivable situation and produce some sort of overall measure, but this would be prohibitive both in terms of time taken and in terms of weighting the different measures. As Elliott says, The problem is that we all think we’re right – well, we know we’re right, the others just think they’re right.