ETHERFABRIC APPROACH
The EtherFabric architecture is the direct result of thinking
outside the box when compared to previous attempts at interconnect
acceleration. The technology is based on the fundamentals of
how the Ethernet-based TCP/IP implementations work but then
takes a different approach to system architecture that produces
significant performance breakthroughs. Increased bandwidth and
significantly lower latency are both accomplished while preserving
Ethernet standards compliance. The EtherFabric approach is illustrated
in Figure 5.
Figure 5 – EtherFabric
architecture
The fundamental breakthrough of this innovative architecture
involves implementing the TCP/IP stack and driver as part of
the user space instead of the kernel space. Thus, we can bypass
the operating system and its kernel, which provides needed latency
benefits but in a manner that is completely compatible. This
allows the hardware acceleration to be both highly efficient
and of modest cost, enabling easy LoM (LAN-on-Motherboard) deployment.
The application API is unchanged from the market-standard Sockets
MPI (and any other) APIs; thus all applications are unaffected,
except for much improved performance! The wire protocol is still
Ethernet, which leverages the $billions of installed equipment
in computer networks, data centers, universities and corporations
worldwide. And the transport protocol is still TCP/IP, bringing
all the reliability benefits developed over the past years of
TCP/IP evolution.
In EtherFabric, Level 5 Networks has created a high-performance
interconnect that:
- Is compatible with existing Ethernet servers,
switches, copper and both long-reach and short-reach optical
physical interfaces, and cables. This contrasts with Infiniband
(and Quadrics and Myrinet), which each require special physical
interfaces, cables, and switches
- Is binary-software-compatible with the vast
installed base of TCP/IP applications and systems software.
This is in stark contrast to iWARP and Infiniband, both of
which use many stacked new protocols, such as RDMA. DDP, and
MPA to implement a reliable transport method different from
TCP/IP.
- Improves the performance of every server into
which it is installed by delivering more CPU cycles to applications,
and requiring far less for network processing.
- Delivers sub-10 usec latency to every connection,
in contrast to conventional Ethernet, which is frequently
50 usec or more.
- Uniquely offers the capability of Single-Sided
Acceleration, whereby connection performance is improved even
if EtherFabric is installed on only one side of a link.
Back to top
The following sections describe previous interconnect approaches,
including key fundamental architectural principles, as well
as the strengths and weaknesses of each.
Conventional Ethernet
The following figure depicts the architecture of the standard Ethernet solution in use today. The major issues that have caused problems for many years are the inefficient use of the host CPU—in some cases 90% of the available processing power can be lost when handling TCP due to context switching and interrupt overhead, cache thrashing and inefficient data handling. These issues also lead to limited throughput, extremely long latencies for applications and have become increasingly problematic as the rest of the equipment in the data center has scaled by orders of magnitude in performance.
Figure 1 – Conventional Ethernet architecture
Back to top
TCP Offload
TCP Offload (see Figure 2) was the first alternative to Ethernet's shortcomings that was explored by the industry. Unfortunately it has fallen short as a solution.
Figure 2 – TCP offload architecture
The premise of TCP Offload is that a coprocessor (a TOE, or TCP Offload Engine) can execute TCP processing faster than the host CPU. However, while this approach solves the CPU loading issue for large data transfers, it does not solve the other critical issue of latency. Additionally, TCP Offload suffers the classic coprocessor problem of not being able to scale up in performance and down in cost through time as quickly as the system processor.
Back to top
InfiniBand Architecture
InfiniBand Architecture (see Figure 3), the next invention
to address the issue, was a clean piece of paper approach. It
addressed all the performance problems of interconnects. It
used an architecture called RDMA (Remote Direct Memory Access)
which bypassed the operating system kernel in delivering data
to applications.
The major drawback of Infiniband is that several new protocols and a new API (Application Programming Interface) were required underneath applications, along with new control plane software. Attempts have since been made to provide shims to map existing APIs to the RDMA protocols, such as SDP (Sockets Direct Protocol), but such efforts often prove to be semantically incompatible with existing applications.Additionally, the performance loss of protocol conversion eliminates the benefit of the new approach. Thus, while Infiniband solved the technical interconnect issues, it was incompatible with existing software infrastructure. This has limited its adoption to arenas where performance gains were so compelling that the software reworking required was considered acceptable. These arenas are limited to large-scale scientific high-performance compute clusters. In other segments, Infiniband has not been accepted and likely will never be now that there is a much easier solution in EtherFabric.
Figure 3 – InfiniBand architecture
Back to top
iWARP
Recognizing the undesirability of requiring all new cabling and switching, iWARP is an attempt to use RDMA with TCP-Offloaded (see Figure 4). The RDMA implementation was borrowed from Infiniband, which gave iWARP the same benefits and drawbacks as the combination of TOE and Infiniband. Namely, latency should be better than TOE-only, but suffers from the same Infiniband issues of a new wire protocol, new incompatible application API and new incompatible control plane software also apply. Additionally, the cost issues seen with TOE approaches are also a factor, and are exacerbated by adding the hardware RDMA engines of Infiniband.
Figure 4 – iWARP architecture
Back to top
