Architecture and Evaluation of a High-Speed Networking Subsystem for Distributed-Memory Sys

Architecture and Evaluation ofa High-Speed Networking Subsystem for Distributed-Memory SystemsPeter Steenkiste, Michael Hemy, Todd Mummert, Brian ZillSchool of Computer ScienceCarnegie Mellon UniversityPittsburgh, Pennsylvania 15213-3891Abstractnetwork standards in various stages of development bystandards bodies. These include ATM (AsynchronousTransfer Mode) [6] and Fibre Channel [26].Achieving high-speed network I/O on distributed-memorysystems is difficult because their architecture is in general Meanwhile, distributed-memory computer systems ill-suited for communication processing. Some of the[2, 11, 13, 15, 24] are becoming the architecture of common problems are: inability to do protocol choice for many supercomputer applications. The reason processing, inefficient handling of data distribution, and is that they are inherently scalable, and provide relatively poor management of the I/O. In this paper we present an inexpensive computing cycles compared with traditional I/O architecture that addresses these problems and uniprocessor or shared-memory multiprocessor supports high-speed network I/O on distributed-memory supercomputers. However, while traditional sequential or systems. The key to good performance is to partition the shared-memory supercomputers such as the Cray have work appropriately between the system and the network been able to make good use of the HIPPI bandwidth [16], interface. We perform some communication tasks on the distributed-memory machines have been much less distributed-memory parallel system since it is more successful. The network interfaces of distributed-memory powerful, and less likely to become a bottleneck than the machines often have low sustained bandwidth, do not network interface. Tasks that do not parallelize well are perform network protocol processing, or manage performed on the network interface and hardware support connections inefficiently. High-speed I/O for distributed-is provided for the most time-critical operations. We memory machines is difficult because the entire system, emphasize the use of simple I/O mechanisms that can be including the architecture, software, programming model used by programming tools that map applications on the and applications, have been tuned to work optimally in a distributed-memory system to implement efficient I/O for distributed fashion, while connecting to a network is an the class of applications they support.inherently centralized and sequential activity.This architecture has been implemented for the iWarpA simple approach to network I/O on distributed-memory distributed-memory system. We describe thismachines is to make the network interface responsible for implementation and present performance results.all network-related processing, so the structure of thenetwork interface will be similar to that of a sequential1. Introductionsystem. However, this places a heavy burden on the Supercomputer applications have to communicate over a network interface since it has to support the I/O of a large high-speed network, for example, to display results on a number (potentially thousands) of processors. The framebuffer, read or write data from storage, or interact network interface can easily become a bottleneck (top with other computer systems as part of a distributed Figure 1), resulting in poor performance.computing application. In the last few years, networksAnother approach is to provide a much simpler network based on the ANSI High-Performance Parallel Interfaceinterface and to minimize the amount of work that is (HIPPI) protocol [12] have become very popular inassigned to it by performing some of the communication supercomputer centers, and all commercially availabletasks on the distributed-memory system itself (bottom supercomputers have a HIPPI interface. HIPPI supports aFigure 1). Earlier research [20] shows that the time spent data rate of 800 Mbit/second or 1.6 Gbit/second. Inon sending and receiving network data is distributed over addition to HIPPI, there are a number of high-speedseveral operations such as copying data, buffer This research was sponsored by the Defense Advanced Research management, protocol processing, and interrupt handling, Projects Agency (DOD) under contract number MDA972-90-and different overheads dominate depending on the C-0035, in part by the National Science Foundation and the circumstances (e.g. packet size). By executing some Defense Advanced Research Projects Agency under Cooperative operations on the distributed-memory system, we can Agreement NCR-8919038 with the Corporation for National reduce the communication software bottleneck on the Research Initiatives.interface. In this paper we present an I/O architecture thatinherently centralized and this creates problems in the following areas:1.Distributed-memory machines achieve their power by distributing the work over a large number of relatively slow processors, but communication protocol processing does not parallelize well.2.The application will often have to manage multiple connections, and this involves scheduling resources in both the distributed-memory system and on the network interface. This is a complicated task and there is a conflict between using general-purpose solutions on one hand, or providing mechanisms that are tailored to specific application on the other.3.Data that is sent or received over the network is typically distributed over the private memories of the nodes. This means that the communication software has to perform scatter and gather operations to collect or distribute the data that makes up the data stream [8]. In networking terms, this is an architecture-specific data transformation that is part of the presentation layer.The three processing tasks that are hard to implement efficiently for distributed-memory systems correspond roughly to the presentation, session and transport layers of the OSI network model (Figure 2).Processor ArrayNetwork InterfaceProcessor ArrayNetwork InterfaceFigure 1:Centralized (top) versus distributed (bottom)communication processing (represented by bullets)takes this approach: it supports high-speed I/O using a simple network interface in a cost effective way. This Application Presentation Session managementTransport protocolNetwork Datalink PhysicalArray Array Interface/Array Interface Interface Interface InterfaceProtocol layers Handled by architecture has been implemented for the iWarp distributed-memory system, and we present performance Figure 2:Mapping of protocol stackresults for this implementation throughout the paper.An additional problem is that on many of today’s The remainder of this paper is organized as follows. In distributed-memory systems, internal links are slower Section 2 we describe our architecture for I/O in than the link speed of high-speed local-area networks distributed-memory systems. We then give a brief such as HIPPI. For example, the peak throughput over overview of iWarp (Section 3) and we discuss our each link in the Intel iWarp and Delta interconnect is 40implementation of the different components of the I/O MByte/second, compared to the 100 MByte/second HIPPI architecture for iWarp: protocol processing (Section 4),link speed. To achieve high sustained network session management (Section 5), and data partitioning throughput, it is necessary to send data between the (Section 6). We summarize in Section 7.system and the network interface over multiple links at the same time. This striping requirement further adds 2. General approachcomplexity to the I/O process.High-speed I/O for distributed-memory machines is In our I/O architecture, as many communication tasks as difficult because the entire system, including the possible are performed on the distributed-memory system,architecture, software, programming model and and the network interface is a relatively simple system applications, have been designed to work in a distributed that is optimized to deal with tasks that do not parallelize fashion, and any form of serialization is avoided if well. Specifically, we map the communication functions possible. The connection to a network, however, isin the following way:1.Transport and network layer protocol processing is 3. iWarp overviewperformed on the network interface, but hardwaresupport is provided for time-critical tasks such as iWarp is a distributed-memory parallel computing system data checksumming (Section 4). The reason is[2]. An iWarp cell consists of a single-chip iWarp that protocol processing does not parallelize well.processor and a local memory. The iWarp processor2.The network interface presents the user with aintegrates both a high-speed computation and simple I/O model that can be implementedcommunication agent in a single component. The efficiently. This model, called the streamsinterface, allows applications on the distributed-communication agent connect the iWarp cell to four memory system to directly manage multiple neighbors through 40 MByte/second buses; the cells in connections to the outside world (Section 5).the iWarp array are configured as a torus. The3.It is the responsibility of the distributed-memory communication system supports high-speedsystem to combine the data that is distributed overinterprocessor communication for a variety of the private memories of the compute nodes incommunication models, including systolic large blocks that can be handled efficiently by thenetwork interface, and to do the reverse scatter communication and memory communication (e.g.operation on receive (Section 6). The reason is conventional message passing as found in distributed that the creation of messages, i.e. reshuffling of memory machines) [3]. Memory communication is data, is a highly parallel operation that can besupported through the use of DMA-like engines that move performed efficiently on distributed-memorydata between the local memory and the interconnection systems.network.We claim that this distribution of tasks between thenetwork interface and the system allows us to supportThe iWarp system software includes optimizing high-speed I/O in a cost effective way.compilers for C and FORTRAN and a cell runtime systemsupporting systolic and memory communication. Users One of the characteristics of the above mapping of taskscan write parallel programs directly for iWarp, but for is that it scales appropriately with increased datamany applications, users rely on parallel program complexity and network speed. Protocol processing, thegenerators when using iWarp. Program generators overhead of which is proportional to the network speed, istranslate a sequential user program into a program for performed on the network interface, while dataeach cell in the system, performing communication and management, the complexity of which depends on thecomputation concurrently on individual cells to achieve application, is done on the same system that performs theadditional efficiency. Program generators can be application.application-specific (e.g. Apply (image processing) Applications on distributed systems can be very diverse[9] and Assign (signal processing) [17]), or more general with respect to I/O. For example, they use different data(e.g. HPF [23]).distributions, and have to interact with a wide range ofexternal devices (displays, disk arrays, other computers).I/O are likely to be inefficient, both in terms ofperformance and resource utilization. Instead, we areproviding a number of simple mechanisms to deal withthe communication processing problems and applicationscan use these mechanisms to implement I/O efficiently.Distributed-memory systems are often programmed usingprogramming tools that map a sequential program on thesystem. These tools manage the resources in the systembased on an understanding of the characteristics of theapplication and distributed-memory system, and they arein the best position to manage the I/O using themechanisms described in this paper.This architecture has been implemented for the iWarpsystem and a HIPPI network by Network SystemsCorporation (NSC). The transmit board has been in use Figure 3:Connection of interface cells to iWarp array by iWarp applications to send data to a HIPPI framebufferiWarp systems communicate with the outside world since spring 93. The full interface was delivered in earlythrough I/O nodes that are linked into the torus at the 1994. UDP over IP is operational, and the TCP port is‘‘edge’’ of the array. Figure 3 shows an example of an nearing completion. NSC also built an interface for DECI/O node (Sun Interface Board, or SIB) linked into the workstations based on the same architecture as the iWarparray in one dimension, and two I/O nodes (HIPPI interface; it connects to the TURBOchannel [7].Interface Boards, or HIB) that are linked into the arrayusing 4 buses. iWarp applications perform output bysending data over the internal interconnect to the I/O of the transport protocol processing, but provides node, which forwards it to the external device, such as a hardware support for per-byte operations.network or disk. Input follows the inverse path. Thisapproach to I/O is very common, e.g. the NCube [10], and the Intel iPSC [15] and Paragon machines follow the same approach.4. Transport protocol processing Transport protocol processing (e.g. TCP or UDP over IP) is one of the potential bottlenecks for high speed network I/O for distributed-memory systems, and it is tempting to try to distribute this operation. However, this is very difficult since most of the protocol processing functions are inherently sequential. There is potential for parallelism between transmit and receive processing (if one is transmitting and receiving at the same time), and ACK and data processing can sometimes proceed in40 MBytes/secparallel [18], but overall, useful parallelism is limited.For these reasons, it is desirable to have protocol Figure 4:HIB block diagram processing performed in a central location, i.e. theFigure 4 shows the architecture of the iWarp-HIPPI network interface. A number of distributed systems use anetwork interface, or HIPPI Interface Board (HIB). It similar approach [19].consists of a transmit and a receive half. Each half One protocol processing task that does parallelize well is includes an iWarp processor that links the network the checksum calculation for the Internet protocols, and it interface into the iWarp torus. The iWarp processor could be performed efficiently on the distributed-memory performs protocol processing, while a ‘‘communication system. However, calculating the checksum on the acceleration block’’ (CAB - shaded area in Figure 4)) system results in an odd ordering of the checksum provides support for critical protocol processing calculation relative to protocol processing. Ideally, the operation: data transfer, checksumming and buffering. checksum calculation on transmit should take place after The architecture uses two iWarp processors instead of one the protocol has broken up the data stream in packets, and because of data bandwidth requirements. The critical on receive, the calculation should take place before resource in the architecture is the memory bus of the protocol processing is invoked since corrupted packets iWarp processors since all data that is sent/received has to have to be ignored. Performing the checksum calculation flow over it, and the bus is also used for program and on the system results in checksumming being performed local data accesses. Using two iWarp processors instead too early on transmit and too late on receive. Although it of one doubles the bandwidth available for these is possible to deal with this, it would add complexity to operations from 160KByte/sec to 320KByte/sec.the software, and for this reason we perform theThe data flow on transmit is as follows: the distributed-checksum calculation on the network interface.memory system forwards the data over the iWarp torus to The operations associated with protocol processing fall in the network interface and it is placed in staging memory two categories: overheads associated with every packet using the iWarp DMA engines. Once a block of data is sent over the network and overhead that scales with the ready to be transmitted, the iWarp processor issues a number of bytes sent. As networks get faster, the per-byte‘‘send’’ call similar to the Unix socket ‘‘write’’ call. This overheads, i.e. data copying and checksumming, become invokes the TCP/IP processing code which performs the dominating overheads, both because the other packetization, and asks the CAB to DMA the data into overheads are amortized over larger packets and because network memory. The checksum is performed in these operations make heavy use of a critical resource: the hardware as part of the data transfer. The packets remain memory bus. The key to making these operations in network memory until they have been acknowledged efficient is to streamline the dataflow [22], i.e. avoid by the destination, so the CAB functions as a retransmit unnecessary copying of data, and to calculate the buffer. The data flow on receive is exactly the inverse, checksum while data is being copied. The per-packet and network memory holds incoming data until the overhead includes time spent in the TCP/IP code, plus application on the distributed-memory system is ready to overhead associated with buffer management, interrupt receive it.handling, and context switching. Careful implementationThe staging memories are dual-port RAMs with a of these operations, as discussed below, should make theirbandwidth of 160MByte/second on the iWarp side and cost acceptable. Based on these observations, the network100MByte/second on the network side. The main role of interface for iWarp relies on iWarp processors to do mostthe staging memories in the architecture is to efficiently gather data coming from the different iWarp buses (transmit) or to scatter data (receive). Since the DMA engines on the iWarp chip interleave the data on the iWarp memory bus in small blocks (8 bytes), it is necessary to use static RAM to achieve high throughput.As a result, the staging memories are small: 128KByte for each direction.The operation of the network interface is similar to that of a sequential system, except that the data source and sink is the distributed-memory system, and not the iWarp processors on the interface. The iWarp nodes on the network interface effectively represent the iWarp array on the HIPPI network.The implementation of the TCP/IP protocol stack for iWarp differs from a traditional workstation implementation in a number of ways. The most obvious difference is that processing is distributed over two processors, so the transmit and receive components of TCP/IP have to be separated. A shared memory (Figure Packet size in Byte5121024204840968192163843276865536T h r o u g h p u t i n M B y t e /s e c o n d1020304050607002564) allows the two components to keep a consistent Figure 5:Bandwidth from network interface to networkprotocol state. Second, the protocol stack has to be as a function of packet size modified to make use of the outboard storage and checksum calculation. These changes are similar to the over the network. The data transfer over the network is protocol stack modifications needed on workstations that controlled by the communication protocols, as described have outboard storage and checksum support; they are above. However, the timing of the data transfers between briefly discussed in [21].the distributed-memory system and the interface cell is critical to the performance of the application, and should Finally, iWarp cells have a very minimal runtime system,be under the control of the application. For example,and several facilities that are available in most operating incoming data should not be transferred to the system systems, such as buffer management, and that are needed before the application is ready to use it since it will by protocols, are not available. This increases the compete with the application for bandwidth on the implementation effort since these facilities have to internal interconnect. It also has to be possible to control implemented. However, it has the advantage that we can the relative order of data transfers over different use optimized implementations, and can avoid a lot of the connections. If the order is different from what the overhead introduced in protocol implementations in application expects, this could result in inefficiency (e.g.existing operating systems. By using an optimized an extra buffering step), or in the extreme case, in UDP/TCP/IP implementation [5] and efficient buffer deadlock.management, we are able to keep the overhead of protocol processing within acceptable bounds.Managing the I/O is mainly a resource management problem: both the distributed-memory system and the Figure 5 shows the bandwidth we can achieve from the network interface have limited resources (memory, buses,interface board to the network. Measurements for raw ..), and how they are allocated to support I/O will have a HIPPI and UDP over IP give the same throughput results,significant impact on performance. On sequential i.e. the UDP/IP implementation is very efficient. The systems, this task is traditionally performed by the main difference between the two cases is that the idle time operating system. On distributed-memory systems, there on the iWarp cell is lower in the UDP case (40%-56%)are a number of ways to handle this task.than in the raw HIPPI case (74%-79%). The current bottleneck in the system is the microcode on the CAB: it A first solution is to have the I/O managed on the front-limits us to sending about 3000 packets per second. We end of the distributed-memory system, typically a Unix expect this number to go up after optimization.workstation: it sets up each communication operation on the network interface. This approach is attractive since 5. The streams I/O modelthe management is done on a general-purpose system, but it has the disadvantage that it adds overhead, since every The transfer of data between the application on the system communication operation has to go through the front end.and the network is a two phase process. In a first phase This will only result in high throughput for very large data (transmit), data is transferred from the system to the transfers. This approach is used for the HIPPI interface ofinterface cell, and in the second phase, the data is sentthe CM2.The stream manager, a system program running on theHIB, uses the format information to manage the data flow A more attractive solution is to have the transfers between the iWarp torus and the network connection ascontrolled directly by the system. However, providing a efficiently as possible. Applications can supply thegeneral-purpose I/O function is both undesirable and specification of data transactions over the streams atunnecessary. First, applications on distributed-memory application startup time or at runtime.systems have very diverse I/O requirements. Therequirements differ in how much data is communicated,Figure 6 shows an example of a single stream that uses athe data format and the timing requirements. Providing a single iWarp pathway. Blocks of data (ADUs) aregeneral-purpose I/O function, specifically for exchanging transferred from the pathway into staging memory, fromdata between the system and the interface, is likely to where they are transferred over the network using theeither have low performance or to consume too many CAB. In the figure, ADUs are transferred through stagingresources. For example, a general-purpose I/O function memory (using double buffering - not shown) one at acould use links in the internal interconnect that are time. If ADUs are small enough, the stream manager willalready heavily used by the application, while other links transfer multiple ADUs at a time to minimize overhead.are idle. Second, distributed-memory machines often are For example, with 16KByte ADUs, 4 ADUs will beprogrammed using higher-level programming tools that transferred at a time resulting in 64KByte writes to themap a sequential program onto the system. These tools network; with double buffering this would utilize thecontrol the resources, since their understanding of both entire 128KByte staging memory.the application and system characteristics allows them touse the resources efficiently, and they are the naturalentity to control I/O.In our approach, the network interface implements asimple I/O model, called the streams model, and theapplication code executing on the system uses this modelto control I/O. The application is responsible for the datatransfer between the array and the HIB. A stream is asequence of bytes that is sent from the iWarp torus to asingle network destination, or from a single networkFigure 6:Example of single stream using one pathway source to the iWarp torus. Applications will in generalrequire multiple streams, and the HIB can support Figure 7 shows an example of a single stream that usesmultiple streams, either simultaneously, or interleaved.four pathways. The stream manager builds ADUs instaging memory by DMAing 1 block from 4 different The stream interface allows the programmer (orpathways. The data organization on the iWarp side is programming tool) to define a stream. This definitionclearly more complicated (see next section) but this includes the source or destination on the network, theapproach makes it possible to achieve throughputs higher protocol used (TCP or UDP over IP, or raw HIPPI), plusthan the internal link speed. Again, the stream manager the definition of one or more data transfers. A datawill try to group ADUs to improve performance. transfer is defined by:•The format of the data on the iWarp side. This is defined in the form of an application data unit (ADU), a block of data that is contiguous in the data sequence on the network, and that is also communicated with the iWarp array as a unit. ADUs can be striped across multiple pathways to achieve a network throughput higher than the internal link speed of the distributed-memory system. The specification of an ADU includes the block size and pathway(s) used.•The format of the data on the network side. If the reliable stream protocol is used (TCP/IP), the data format on the network side is a byte stream, and noformat specification is needed; packetization andheader creation are done by TCP. If the datagram Figure 7:Single stream striped across 4 pathways protocol (UDP/IP) or raw HIPPI is used, then theuser is responsible for packetization, and the user Applications will sometimes need several streams. has to specify the message size and (optionally) a Typically there will not be enough resources on the HIB header for network communication.to have all streams operate efficiently at the same time,•Mapping between the format of the data streameither because there are not enough DMA engines in the over the network and iWarp. This corresponds toHIB iWarp cell, or because the staging memory would specifying the order in which data travels over thenetwork.become too fragmented. The solution is to have only asmall number (possibly one) of active streams . These streams have resources allocated to them, and the stream manager controls their data flow. The other streams are idle, waiting for resources. The unit of interleaving, i.e.how long a stream remains active, is defined by the application, and will typically have some relationship to the structure of the application data.6. Data reshuffling in support of high-speed I/OTo efficiently utilize the large number of processors in aiWarp arrayMatrix distributed-memory computer, applications typically use (a) Row-swath partitioning over 4 by 4 arraydata parallelism. Data is partitioned into equal-sized blocks, which are distributed across the processors, and each processor operates on the data that is assigned to it.Both the type and granularity of data partitionings varies widely between application. As a result, I/O operations include an extensive scatter/gather operation that is application specific. The scatter/gather operation also has to deal with striping the data stream over multiple buses to achieve higher throughput, as was discussed in the previous section.In this section we first describe some commonly usediWarp arrayMatrix data partitionings, and we then discuss how data (b) Coarse-grain block partitioning over 4 by 4 arraypartitioning can be handled efficiently.6.1. Data partitioningFigure 8 shows some of the mappings that are used by iWarp applications. The row-swath partitioning is used by the Adapt image processing environment [25]. The coarse-grain block partitioning is used by several image processing applications and fine-grain block partitioning is used in the iWarp implementation of the LAPACK library [14]. All three examples are instances of one or two dimensional block-cyclic partitionings.iWarp arrayMatrix (c) Fine-grain block partitioning over 2 by 2 array I/O of a distributed data structure becomes harder as the partitioning is finer. A good measure of the granularity is Figure 8:Common data mappings on iWarp the size of blocks of data that are contiguous both on the network and in the system: when a data structure is partitioned in smaller blocks, the task of gathering the data for transmission over the network will be more significant. Table 1 shows the block size and number of blocks for the partitionings of Figure 8 for an N ×N matrix mapped on a p ×p processor array, using a blocking factor of b for the fine-grain mapping; we assume the array is sent over the network in row order. A specific example is given in Table 2. We see that the block sizes are quite small for the 2D partitionings.Table 1Granularity of 3 common mappings 6.2. StrategyDistributed-memory machines can deal with the data interconnect is small, the interface will become a distribution in a number of ways. In a first approach,bottleneck for fine-grain data partitionings.array cells send data to the interface independently and In the approach we selected, the processor array is the data is sorted out and grouped by the interface. This responsible for the scatter/gather operation. On transmit,approach is simple to implement, but even if the overhead it constructs large messages and presents them to thefor sending and receiving blocks to or from the internal。