Implementing Virtual Shared Memory Libraries on MPI

Implementing Virtual Shared Memory Libraries on MPIJ. Carreira, L.M. Silva, J.G. SilvaUniversity of Coimbra, Portugal Email: {jcar, lams, jgabriel}@mercurio.uc.ptSimon ChappleEdinburgh Parallel Computing Centre, UKEmail: simonc@ AbstractParallel programming using explicit message passing has become widespread mainly due to performance and portability reasons. However, there are many algorithms for which the message passing implementation is non-trivial, tedious and error-prone. A possible solution is to use higher level programming models that facilitate the task of programming in parallel. Portability of these higher level programming paradigms can be achieved if they are implemented as parallel libraries on top of a standard message passing environments like MPI.This paper describes two projects that were developed with the aim of providing virtual shared memory programming abstractions on top of distributed memory machines. Two libraries were implemented on top of CHIMP/MPI: Eilean and DSMPI. The interface and functionality of the libraries are briefly presented and the shortcomings and nice features of MPI for the development of parallel libraries discussed. The discussion is focused on issues such as heterogeneity, multithreading, and dynamic process creation and is particularly directed to libraries that are based in the concept of data servers such as DSMPI and Eilean.Finally, we hope that the points herein discussed can lead to a better understanding of the current limitations of MPI and be useful in the preparation of the forthcoming MPI-2 standard.1. IntroductionThis paper describes two projects that were developed with the aim of providing virtual shared memory (VSM) programming abstractions on top of distributed memory machines. Two libraries were implemented on top of MPI: Eilean and DSMPI.Eilean is a library that follows the Linda parallel programming paradigm. Linda is based on the abstraction of the Tuple Space, an unordered bag of tuples which is accessed by all processes enrolled in the computation and is content addressable like an associative memory. Eilean proposes some enhancements concerning former Linda implementations. It allows the programmer to specify expected tuple access patterns in order to select the most appropriate policy for tuple distribution.DSMPI is a structure-based Distributed Shared Memory library. It also provides different distribution protocols depending on the characteristics of the shared data object and its access pattern. It implements three different models of consistency: sequential consistency, release consistency and lazy-release consistency.While there is an inherent performance penalty in using high level libraries instead of MPI, the gains in terms of code reuse and programming effort can be considerable [Clarke94]. Bothlibraries try to reduce the performance penalty of an extra layer on top of MPI by using flexible data distributions that can be selected by the user.The two libraries have some design aspects in common: they contain data servers which are responsible for the management of shared data elements (either tuples in Eilean or traditional data-structures in DSMPI) and a run-time system that implements the protocols to guarantee the consistency of distributed data.We start the paper by briefly describing the programming interfaces and the basic features of the libraries in the next section. Afterwards, we discuss the shortcomings of MPI that rise some important problems in the development of this kind of client-server structure and the important decisions we made to turnaround those limitations and use MPI as efficient as possible.It is worth to note that the issues discussed in this paper are not particular to Eilean and DSMPI. Rather, they are common to any high-level programming library that is based on the concept of data servers. This is particularly true for issues such as heterogeneity, multithreading servers, active messages and dynamic process creation.2. Virtual Shared Memory LibrariesIn this section we give a brief description of the two parallel libraries: DSMPI and Eilean.2.1 DSMPIDSMPI is a parallel library implemented on top of CHIMP/MPI [CHIMP] that implements the abstraction of distributed shared memory. The most important guidelines that we took into account during the design of the library were:1. assure full portability of DSMPI programs;2. provide an easy-to-use and flexible programming interface;3. support heterogeneous computing platforms;4. optimize the DSM implementation to allow execution efficiency.The library was implemented without using any special features of the underlying operating system. It does not require any special compiler, pre-processor or linker. This is a key feature to obtain portability. In this point, we depart from previous approaches of DSM implementations [Nitzberg91].All shared data and the read/write operations should be declared explicitly by the application programmer. The sharing unit is a program variable or a data structure. For this reason, DSMPI can be classified as a structure-based DSM system (like ADSMITH [Liang94]). Other DSM systems that are implemented in a page basis are prone to the problem of false sharing [Li89]. DSMPI does not incur in this problem because the unit of shared data is completely dynamic and related to existing objects (or data structures) of the application. At the same time, it allows the use of heterogeneous computing platforms since the library knows the exact format of each shared data object, while most of the other DSM systems are limited to homogeneous platforms. Finally, DSMPI allows the coexistence of both programming models (message passing and shared data) within the same application. This has been considered recently as a promising solution for parallel programming [Kranz93].Concerning absolute performance, we can expect applications that use DSM to perform worse than their message passing counterparts. However, this is not always true. It really depends on the memory-access pattern of the application and on the way the DSM system manages the consistency of replicated data.In DSMPI we tried to optimize the accesses to shared data by introducing three different protocols of data replication and three different models of consistency that can be adapted to each particular application in order to exploit its semantics. With such facilities we expect DSM programs to be competitive with MPI programs in terms of performance. Some performance results collected so far corroborate this expectation.2.1.1 DescriptionIn DSMPI there are two kinds of processes: application processes and daemon processes. The latter ones are responsible for the management of replicated data and for the protocols of consistency. Since the current implementations of MPI [MPI] are not thread-safe we had to implement the DSMPI daemons as separate processes. This is a limitation of the current version of DSMPI that should be relaxed as soon as there is some thread-safe implementation of MPI. All the communication between daemons and application processes is done by message passing. Each application process has access to a local cache that is located in its own address space and where it keeps the copies of replicated data objects. The daemon processes maintain the primary copies of the shared objects. DSMPI maintains a two-level memory hierarchy: a local cache and a remote shared memory that is located in and managed by the daemons.The number of daemons used by an application is chosen by the programmer as well as its location, by using a configuration file (dsmconf) that is read by the DSMPI initialization routine. This approach gives to the user the freedom to choose the most convenient mapping of daemon processes according to the application needs.The ownership of the data objects is implemented through a static distributed scheme. While a centralized scheme would introduce a bottleneck, a dynamic distributed policy would require the use of broadcasts or forward messages to determine the current owner of a data object [Stumm90]. In the static distributed policy, the owner daemon of each object is chosen by the run-time system during the startup of the application and remains fixed during the lifetime of the application. Each process maintains a local directory containing the location of each object in the system. This static distribution strategy requires less control messages than the dynamic strategy and does not introduce the congestion of a central server.2.1.2 Data Replication ProtocolsThe library allows the programmer to chose the replication strategy for each shared object among three possible choices. Shared objects can be classified as:1. WRITE_MOSTLY;2. READ_WRITE;3. READ_MOSTLY.The first class uses a single copy of the object while the two other classes replicate the object. Data replication is one way of exploiting parallelism, since multiple reads can be executed in parallel.The first class (WRITE_MOSTLY) represents those objects that are frequently written. For such objects is not worthwhile to replicate them among the caches of the processes. Only one copy is maintained by one of the daemons.The second class (READ_WRITE) comprise those objects that have roughly the same number or read and write request. These objects are replicated among the caches of the processes that perform some read request on them. There is one daemon that keeps the primary copy of the object and maintains a copy-set list of all the processes that have a copy in its local cache. A process is included in that list when it issues a remote read request. After that and in the absence of write operations, the process reads the object from its local cache. That daemon is responsible for the consistency of the replicated object, and for this particular class the library implements a write-invalidation protocol. This means that on a write operation all the cached copies of the object are invalidated from the local caches. Only the process that writes to the object maintains an updated copy in its local cache. When some of the other processes wants to read or write that object it gets a cache-miss, and it has to fetch the object remotely from the daemon that maintains the primary copy.The objects that belong to the third class (READ_MOSTLY) are also replicated among the caches of the processes that use them. This objects have a higher ratio of read over write requests and for this class the library uses a write-update protocol. All the cached copies of the object are updated atomically after each write operation. This scheme does not incur in cache-misses as the write-invalidation protocol but in the case of large size objects it requires the sending of large size messages that carry the update notices.Both protocols have advantages and disadvantages [Stumm90]. Invalidation-based protocols have the advantage of using smaller consistency-related messages than update-based protocols since they only specify the object that needs to be invalidated and not the data itself. The write-update protocols reduce the likelihood of a cache-miss.The application programmer has the freedom to choose between those three classes when it creates a particular shared object. It has been argued in [Stumm90] that no single algorithm for DSM will be suitable for most applications. Shared objects have different characteristics, thus it is important to select the coherence protocols that best match the application characteristics.2.1.3 Models of ConsistencyIn order to exploit further efficiency, we have implemented three different models of consistency:1. Sequential Consistency (SC);2. Release Consistency (RC);3. Lazy Release Consistency (LRC).The SC model was proposed in the IVY system [Li89], where all the operations in shared data are performed as if they were following a sequential order. This is the normal model of consistency, that in some particular cases does not perform very well in DSM systems implemented on networks like Ethernet.For such cases, the programmer can make use of the RC model, that in our case implements the protocol of the DASH multiprocessor [Lenoski90]. In this model, it is assumed that all the shared accesses are protected by synchronization primitives (acquire and release,corresponding to lock/unlock operations). It attempts to mask the latency of write operations by allowing them to be performed in the background with the computation. The process is only stalled when executing a release, at which time it must wait for all its previous writes to perform. However, this model does not attempt to reduce the high cost of moving data across the system. It only tries to reduce the latency of write operations.To reduce the message traffic the programmer can use the LRC model of consistency. DSMPI implements a similar protocol as proposed in the TreadMarks system [Keleher94]. This model relaxes the Release Consistency protocol by further postponing the sending of write notices until the next acquire. When a process writes into a replicated data object, rather than sending update/invalidate notices to all potentially interested processes (i.e. those that belong to the copy-set list), it keeps a record of the objects that it updated. Then the releaser notifies the acquirer of which objects have been modified, causing the acquirer to invalidate its local copies of these objects.The LRC scheme keeps track of the causal dependencies between write operations, acquires and releases, allowing it to propagate the write notices lazily only when they are needed. To implement this scheme we have used the Vector Time mechanism, and the resulting protocol, albeit complex, reduces considerably the number of messages and the amount of data exchanged on behalf of the replication protocols.2.1.4 InterfaceThe library provides a C interface and the programmer calls the DSMPI functions in the same way it calls any MPI routine. The complete interface is shown in Figure 1.- DSMPI_Startup(MPI_Comm *comm);- DSMPI_Exit(void);- DSMPI_Decl_SO(char*name, int count, MPI_Datatype type, DSM_SO *shobj);- DSMPI_Create_SO(DSM_SO shobj, int access_type);- DSMPI_Read(DSM_SO shobj, void *buf, int model);- DSMPI_Write(DSM_SO shobj, void *buf, int model);- DSMPI_ReadSome(DSM_SO shobj, void *buf, int count, int offset, int model);- DSMPI_WriteSome(DSM_SO shobj, void *buf, int count, int offset, int model);- DSMPI_Init_Lock(int lock_id);- DSMPI_Lock(int lock_id);- DSMPI_UnLock(int lock_id);- DSMPI_Init_Sem(int sem_id, int value);- DSMPI_Wait(int sem_id);- DSMPI_Signal(int sem_id, int N);- DSMPI_Barrier(int barrier_id, int N);Figure 1: DSMPI primitives.Initialization and TerminationTo start a DSMPI application all the processes must invoke DSMPI_Startup() collectively. This function returns a MPI communicator that can be used by the application processes to communicate directly through message passing. To terminate correctly a DSMPI application all the processes have to call DSMPI_Exit().Declaration and Creation of Shared ObjectsThe set of shared objects used in the application should be declared explicitly by all the processes at the beginning of the application through the routine DSMPI_Decl_SO(). With this routine the programmer has to specify the data type of the object, and the number of data elements. It returns a shared object identifier that should be used in any further reference to that object. To create one shared object a process has to call the DSMPI_Create() primitive. In this call, the programmer has the freedom to chose the replication strategy for the shared objects by indicating the access_type among the three options: WRITE_MOSTLY, READ_WRITE and READ_MOSTLY.Read and Write OperationsCommunication between processes is made by using explicit read and write operations on shared objects. The programmer just calls DSMPI_Read() or DSMPI_Write() and does not need to be aware of the location of the object.In the current version, the programmer has the freedom to specify the required model for data consistency among three choices: DSM_SC, DSM_DRC and DSM_LRC, where DSM_SC corresponds to Sequential Consistency, DSM_DRC corresponds to the Dash Release Consistency protocol, and finally DSM_LRC corresponds to Lazy Release Consistency.To read or write only parts of some object the programmer may find the following routines very useful: DSMPI_ReadSome() and DSMPI_WriteSome(). The programmer has only to specify the offset within the object and the number of basic elements he wants to read (or write). These routines have been proved to be very effective to operate in large size objects (like arrays and vectors) when the program only wants to read or update a small part of the object.Synchronization PrimitivesTo control the concurrent access to shared variables the programmer has to use some synchronization primitives. DSMPI provides locks (DSMPI_Lock(), DSMPI_UnLock()), semaphores (DSMPI_Wait(), DSMPI_Signal()) and barriers (DSMPI_Barrier()).These are the usual primitives provided by most DSM systems. More details about DSMPI can be found in [Silva95].2.2 EILEANThe Linda parallel programming paradigm [Ahuja86] is a very popular model for parallel programming due to its simplicity and elegance. It is based on the notion of the Tuple Space (TS), a logically shared memory which basic data unit is the logical tuple, and may be accessed by all the processes.Linda has become widespread and was implemented in shared memory as well as in distributed memory machines. However there are several obstacles to produce a portable, scalable and efficient Linda system which can run on large distributed memory machines, and in Workstation clusters.For a Tuple space implementation to be efficient, the underlying pattern of messages used to support TS operations for a particular Linda program should closely resemble the pattern of messages for a message-passing implementation of the same program.Eilean is a parallel library for MPI based on the Linda programming paradigm, that intends to provide a scalable distributed Tuple Space using tuple type specific distribution policies.Tuple Space scalability is achieved by making the number of Tuple Space servers grow with the number of processors. The configuration of the Eilean servers among the available machines can be manually selected by the programmer, or automatically by the Eilean run-time system. The rationale of the first approach is to let the user to take advantage of different node connections with different speeds, common in large heterogeneous environments, and distribute the servers accordingly to the particular characteristics of his/her system.The goal of this distribution is to make a static, yet general partition of the Tuple. With such general structure, the run-time system, aided by programmer hints, can map tuples close to the processes which use them, optimize tuple movements and reduce the overhead of Linda operations.2.2.1 InterfaceFor an MPI process to enroll in the Eilean environment it must invoke EileanStartup(). This call initializes library internal data and establishes communication with the TS servers. Only after this call the application can execute the usual Linda primitives.To leave the Eilean environment the processes have to call EileanTerm(). After this call, the MPI processes cannot execute any more Linda primitive. However it is worth to note that, while enrolled in Eilean, processes can still communicate using MPI primitives and can continue processing normally after leaving Eilean.This is extremely important as the programmer can use message passing directly to solve some specific task within the program.Tuples are represented in Eilean only by an integer (first field) and an MPI datatype (basic or derived). This simplification concerning the original Linda was driven by the fact that the focus of our study is on tuple distribution and access strategies. However the complete Linda interface with tuples of arbitrary size and complete matching will be added in a future version if the results prove to be satisfactory. Actually, tuple matching is restricted to the first integer field of the tuples.Eilean provides the usual Linda primitives to access tuples: in(), out(), rd(), eval(), rdp() and inp().Each require three parameters: an integer tag, a pointer to a variable and the correspondent MPI_Datatype handle.2.2.2 Programmer HintsTo manage each tuple in the most appropriate way, Eilean treats tuples differently according to their use within the application. The information about tuple access patterns is gathered during run-time and explicitly given by the programmer.In most applications, the programmer has a detailed knowledge of the tuple usage patterns, and it is important that such knowledge can be given to the system. The run-time system use this information to determine the best distribution policy to be used with each tuple type, and map it accordingly in the available servers. We distinguish four general Tuple classes according to its access patterns:1. Write-many tuples;2. Producer;3. Consumer;4. Read-most tuples.Write-many tuples are frequently modified by multiple processes. This kind of objects is problematic in any parallel system. The best solution seems to keep them centralized in a TS server. Eilean selects the servers to house tuples in this category by applying an hashing function based on the overall number of servers to the first integer tuple field. Tuples of this kind, will not be stored all in the same server but distributed according to its first field. It is worth to note that this policy would be very inefficient if applied to all tuples, but in Eilean it is applied just for a minor subset of tuples for which there is no other pattern to exploit.Producer and Consumer tuples are by far one of the most common in Linda. They are generated by one process and withdrawned by another(s) in a fixed pattern. This is, for instance, the case of job tuples in the Master-worker paradigm. They are handled performing eager tuple movement, i.e. migrating tuples to the processes where they will be consumed. To give this hint to the Eilean run time system the programmer simply has to invoke the Hint() function call in the client process specifying that it is willing to consume a certain type of tuple. For instance:Hint(tag, CONSUMER);Conversely, the producer of the same tuples should inform the run time system that it is willing to produce those tuples, therefore the following hint should be given: Hint(tag, PRODUCER);Result tuples are generated by workers and consumed by the master. They are simply handled the opposite way, i.e. declaring the master as consumer and the workers as producers. When a process out’s a tuple marked as producer, instead of sending the tuple to the TS, the process tries first to send it in advance, directly to a consumer where it will later be in’ed. The list of consumers in the correspondent producer processes is managed and updated by the run time system. To accomplish this, Eilean uses MPI synchronous non-blocking sends: when a process tries to out a tuple and its known consumers have not in’ed the previously send tuples yet, they are send to the hashed TS server. Finally, this server uses again synchronous non-blocking sends to send the tuple in advance to any of its consumers and polling to guarantee that they are feed constantly.Read-most tuples are read far often than they are written. This is the case of general counters or reference values, and can be identified as tuples that are most accessed by Rd requests. These tuples are centralized in TS servers determined by hashing the first field, but a copy is also kept in the local tuple cache of each process that declares itself as a reader. The declaration is made through the Hint function in the following way:Hint(tag, READ_MANY);Each TS server manages the READ_MANY tuples that are housed in it and keeps the consistency of the copies by sending invalidate and update message to the readers whenever necessary as a consequence of in() or out() operations. For this optimization to be effective, we also include two new Linda operations, Upd() and Add() [Carreira94].2.2.3 Dynamic Process CreationOne of the TS servers, is called the Eval server and also acts as a global coordinating manager which supervises the launching of processes through the Linda operation eval().At startup, all MPI processes are blocked waiting for an eval request except the main Linda program. Whenever an eval is issued, a request is send to the eval server that keeps track of the nodes already running Linda processes and decides in which node it should be started or if it should be queued until a processor becomes available.The eval server resembles a mechanism of dynamic process creation on top of MPI, however there are no real processes being launched. The current MPI philosophy is SPMD oriented and all processes are launched during startup. In Eilean, these MPI processes simply block on startup waiting for an eval request (inside EileanStartup()) and when it arrives call the user C function identified in the request. Afterwards, the C function can perform its task using all the Eilean primitives (including eval()). When the function terminates the eval server is signaled and the node is free to receive another eval request, for the same or other C function of the user application.To create global identifiers for the functions that are launched by eval during the execution of the application, the user must declare them before calling EileanStartup(). This can be made by using an Eilean primitive that receive a pointer to a function as the only argument. For example:DeclareWorker(slave);DeclareWorker(adder);Although, actually this must be done explicitly by the programmer, this is a task that will be handled by a preprocessor in the next version of Eilean.More details about Eilean can be found in [Carreira94].3. HeterogeneityIt is highly desirable to integrate heterogeneous hosts into a coherent computing environment to support parallel applications. Both libraries require from the application programmer the declaration of the data types of the shared objects or tuples. From the moment that the library knows the exact format of each shared data object it becomes quite easy to support heterogeneity in DSMPI and Eilean thanks to the MPI capability to support computing platforms with different data representations.4. MultithreadingA thread can basically be defined as a single flow of control within a process. The primary motivation for the use of threads was the handling of asynchronous independent events within the same process. Additionally, threads can enhance the performance of communicating processes in many situations by masking communication latency. i.e. when computing can be overlapped with I/O, by using two threads one for each task. Also, when processes are frequently created and destroyed within applications the use of threads or lightweightprocesses can significantly increase performance due to the smaller context data and low cost of context switches.An effort towards the standardization of a thread interface was made that resulted in the POSIX 1003.4a thread extension - Pthreads [POSIX]. A wide variety of packages providing the Pthreads interface is available nowadays, such as [Mueller93].The MPI standard was designed to make possible thread-safe implementations, but in fact it does not require that implementations must be thread-safe. Thread safety implies that all functions of MPI should be reentrant, i.e. could be safely invoked by multiple threads concurrently without leading to wrong results or deadlocks. Unfortunately, none of the current implementations of MPI are thread-safe.Multithreading is an issue of paramount importance for the development of data servers on top of MPI. Without it, data servers have to be implemented as separate heavyweight processes what will lead to several inefficiencies:Slower communications: Two processes, even if running in the same machine, have to communicate by using the MPI primitives. In a multithreaded server, accesses from a local client running in the same node can be done using shared memory and thus achieve a better performance. Requests from remote nodes to that server have the same cost as with a heavyweight server.Inefficient caching:As was explained in the previous section, caching can be very effective in increasing the performance of VSM packages. An efficient cache implemented in the client side requires a separate control of the caching protocols and thus a dedicate thread. Otherwise, cache management can only work occasionally when the user application enters the library through any of its calls.Wasting of processors:There are machines with a strict SPMD model such as the CRAY-T3D, where is only allowed one process per processor and all processors are loaded with the same binary. This implies that each data daemon have to occupy exclusively a processor and the application and daemon codes have to be gathered together in the same executable. A multithreaded server would coexist in the same processor as one or more clients.Some efforts have been made to provide a thread-safe implementation of MPI [Chowdappa94] or even going further and integrating MPI and Pthreads [Haines94]. However, apparently several problems have impaired these efforts and there is no thread-safe MPI implementation at the moment. Interestingly, there is already a thread-safe implementation for PVM called TPVM [Ferrari 95]. It would be interesting to see in forecoming version of MPI some support for a threads package like Pthreads.Other benefit that arise from an integration with Pthreads is that MPI implementations could be themselves multithreaded. This would allow the inclusion in MPI-2 of some features like non-collective multicasts based on broadcast trees and efficient implementation of non-blocking communications.5. MPI Implementation DetailsIn several stages of the development of the libraries, the developers felt the need to have more information about the internal details of the particular MPI implementation. For instance, to perform eager movement of data to client processes using immediate sends it is absolutely。