Level 5 Boosts Ethernet Bandwidth, Lowers Latency

by Timothy Prickett Morgan

Nature loves evolution, not revolution, and it is not by coincidence that so do the customers who end up buying (or not buying) the myriad hard and soft wares put together by the vendors in the computer and networking industries. That is why Level 5 Networks, which launched its first product yesterday, is so interesting. Level 5 has taken the ubiquitous Ethernet protocol and created a network fabric that has many of the good attributes of more sophisticated networking while maintaining compatibility with plain, old Ethernet.

Level 5 was created by a bunch of networking experts from the AT&T Laboratories Cambridge, in the United Kingdom, which was a research lab set up by Italian computer and networking equipment provider Olivetti years earlier. After AT&T shut down the Cambridge lab, Derek Roberts, who was the architect of AT&T's CLAN high-performance networking project and who is now Level 5's chief scientist and vice president of hardware architecture, and Steve Pope, who also worked on the CLAN project at AT&T and who is chief technology officer and vice president of software architecture, got together in 2002 to try to fix Ethernet and TCP/IP. While both protocols work fine for relatively modest workloads, Ethernet is too slow for many high performance workloads (such as supercomputer clusters and database clusters), and the TCP/IP stack adds too many latencies and eats up too much server processing power.

According to Dan Karr, who was brought in as president and CEO of Level 5 from semiconductor maker Virata and who was head of sales for graphics chip maker S3 as it grew from $10 million to more than $300 million in sales, there are a number of networking products that have tried to do what Ethernet cannot, which is move lots of data at a low latency with a lot of throughput and a high connection rate. Quadrics and Myricom created high-speed, high-bandwidth interconnections for workloads that use the Message Passing Interface (MPI) protocol common in high-performance supercomputing clusters; the InfiniBand protocol was created to bridge the gap between the supercomputer and commercial worlds and was intended to be a fabric for connecting servers to each other and to their storage; and even the iWarp (InfiniBand over Ethernet) protocol from the RDMA Consortium and the Internet Engineering Task Force requires some changes to the operating system software stack and APIs. Both InfiniBand and iWarp are smart in that they get applications talking to the host adapters and offload some operations from the operating system--which provides reduced latency and very good performance--but they require new APIs. And TCP/IP offload adapters help some on big data blocks, but do not do anything for small blocks of data and they actually increase latencies in the network for some data transfers.

What Level 5 claims to have done with its EtherFabric is to soup up Ethernet and TCP/IP without making operating system or server makers tweak their wares. "A lot of people say the words 'Ethernet compatibility,' but they do not exactly deliver on it," explains Karr. "With EtherFabric, there is no porting, no compilation of applications or operating systems, no change to the network." He says that being able to use the same wires is only 2 percent of the problem, which is what solutions such as iWarp provide. If you have to change your software, saving money on the wires is probably not worth it.

You can bet that supporters of Quadrics, Myricom, InfiniBand, and iWarp are going to come out pretty strongly to pick apart the claims that Level 5 is making with EtherFabric because the technology seems compelling even if it may not offer the lowest latency among interconnections. The company's first product will be a two port, 1 Gb/sec Ethernet card, and it will soon deliver 2 Gb/sec and 4 Gb/sec parts. These are the speeds of Gigabit Ethernet and the two Fibre Channel interconnections. There are a number of secrets to EtherFabric, but basically it has electronic circuits that virtualize the network interfaces and handle much of the work normally done by the operating system on a server that is managing TCP/IP and Ethernet links; then, the TCP/IP stack is moved up into the application user space on the servers and each application gets its own virtual TCP/IP link. TCP/IP exception processing is grabbed by the EtherFabric host adapter and re-routed to the existing operating system kernel so the OS does not have to be changes.

The net effect, says Karr, is that compared to Gigabit Ethernet, twice as many CPU cycles are freed up (which means clusters can have half as many servers), and latency improves by a factor of five to under 10 microseconds (measured at an application-to-application level). And because the EtherFabric hardware and software supports channel bonding (which is striping data across multiple, virtual Ethernet ports), customers who need more bandwidth can gang up both ports on the single EtherFabric card to make a 2 Gb/sec port, or can take multiple cards to create even more bandwidth. What this also means, for instance, is that if you want to use iSCSI to connect to your network storage instead of Fibre Channel, you can do so over port-striped EtherFabric links and have the same bandwidth as the fastest Fibre Channel SAN links available today. This is a big deal for a lot of customers.

While Level 5 has 2 Gb/sec and 4 Gb/sec ports on the way, it says by the first half of 2006, it will have 10 Gb/sec ports available. The EtherFabric cards come in PCI and PCI-X flavors, and will plug into faster PCI Express slots by the end of the year.

One of the interesting aspects of the EtherFabric approach is that if you just plug it into one side of a link, it will speed up that link. You don't have to have it on both sides to improve the performance of a machine, although having it on both sides of a link will obviously be a better idea.

Right now, the EtherFabric hardware and software will support any open source or commercial Linux running the Linux 2.4 or 2.6 kernels. A single two-port, 1 Gb/sec adapter including the software costs $495, with volume customers being able to get them for $295 a pop. Early next year, Level 5 will roll out support for Microsoft's Windows Server 2003 operating system, and can deploy the technology on any Unix platform it wants and will do so based on customer demand. The company is engaged with the major server makers to see how it can work together to push EtherFabric into the market.

Level 5 had last year secured $9 million in venture funding to help build its products, and in addition to announcing the availability of EtherFabric yesterday, Level 5 said it had actually secured a second round of funding amounting to $30 million back in February. Oak Investment Partners, Accel Partners, Amadeus Capital Partners, and IDG Ventures all kicked in the dough, which will be used to build out and market the EtherFabric product line.