Communication support for distributed collaborative applications

Communication Support for Distributed Collaborative ApplicationsRaj Yavatkar and kshmanDepartment of Computer ScienceUniversity of Kentucky,Lexington,KY40506-0027raj,lakshman@To appear in the ACM/Springer V erlag Journal on Multimedia Systems,V ol.2,No.2,1994.July19,1994AbstractThe Development of distributed,multimedia,collaborative applications requires resolution of com-munication issues such as concurrency control,and temporal and causal synchronization of traffic overrelated data streams.Existing transport and/or session layer protocols do not include desired supportfor multi-stream,multipoint communication.In this paper,we propose new communication abstractionsand mechanisms that facilitate implementation of the necessary coordination and concurrency controlsemantics in a collaborative application.We also propose a protocol suite called MCP(Multi-FlowConversation Protocol)for realization of these abstractions and describe its prototype implementation inan internetwork of workstations.The paper also describes our experience with the prototype and resultsof a performance evaluation.KEYWORDS:multimedia communication,distributed collaboration,temporal and causal synchroniza-tion,Unix networking.1Introduction1.1MotivationDistributed collaborative systems allow a geographically dispersed group of users or applications to use computers and multiple media(text,audio,video,and images)to solve problems together.An example of such an application is a collaborative,software engineering environment[SFB87,AEHL88].In such an environment,a group of designers located at different sites collaborate on a design document(or a program) using interactive tools to edit and test parts of the design under development.Interaction may involve a group editor(based on shared windows),an image display(that displays resulting design),and a voice(and video)channel that allows them to view,discuss,and edit the suggestions made by each other.Moreover, the interaction may not necessarily involve human users,but may also involve cooperative programs that work together in automating a particular task[DL87].Development of such collaborative tools requires research in several areas including user interfaces, communication infrastructure,collaboration paradigms,group toolkits,and shared object systems[PD93].In this paper,we focus only on the communication aspects of distributed collaborative mu-nication in such applications involves several,related,multipoint streams.Each stream carries data specific to a particular medium and is shared among several geographically dispersed participants.Because several participants share one or more streams,some form of concurrency control is desired.In addition,data sent over separate streams are typically related in time and context,and such relationships must be preserved in delivery of data at each recipient.Thus,distributed collaboration using multiple media requires resolution of coordination and control issues such as concurrency control,and temporal and causal synchronization of traffic over related data streams.Currently,conventional transport and session layer protocols do not include desired support for multi-stream,multipoint communication.Instead,collaborative application designers must build their own mechanisms to achieve desired semantics.This approach has several disadvantages.First,lack of suitable abstractions requires designers to concentrate on low-level communication aspects instead of focusing their attention on investigating higher-level issues such as collaboration paradigms,flexible toolkits,and user interfaces.Second,application layer solutions for coordination and control must use an existing transport protocol such as TCP(transmission control protocol)or UDP(user datagram protocol)designed only for asingle communication stream between two parties.Therefore,application layer implementations are not efficient and require duplication of some of the transport and session layer functions includingflow and error control,buffer management,asynchronous event handling,and timer management.Third,lack of standard interfaces and abstractions leads to collaborative systems that are not inter-operable.In contrast,distributed applications based on traditional client-server model utilize well-defined com-munication abstractions designed for two-party communication.Necessary semantics are efficiently im-plemented in transport and session layer protocols allowing application designers to concentrate on higher level issues.Clearly,there is a need for similar functionality and support from communication protocols to facilitate development of collaborative multimedia applications.1.2Our ContributionsIn this paper,we investigate design and implementation of suitable communication abstractions and mecha-nisms to allow realization of the necessary coordination and control semantics in a collaborative application. In particular,we have designed a new protocol called Multi-Flow Conversation Protocol(MCP)that provides two communication abstractions:First,MCP includes a token-based mechanism for concurrency control among participants of a multipoint connection.The degree of concurrency control achieved varies from a strict“floor-based”control at one extreme to no control at the other extreme.Second,MCP provides a communication abstraction called a multi-flow conversation to allow temporal and causal synchronization among traffic over multiple, independent streams.A conversation may consist of one or more(two-party or multipoint)connections and provides both temporal and causal synchronization in delivery of messages sent over participant connections. Temporal and causal ordering in a conversation is restricted based on a notion of delay-constrained or∆-causality that takes into account the deadlines on delivery of multimedia data.Additional features of MCP include an out-of-band signaling channel for exchanging control information and an asynchronous interface that allows an application to specify event-triggered actions to facilitate fast processing of periodic data and changes to the status of multipoint connections.MCP is an upper level protocol designed to support a wide variety of collaborative applications.We have implemented it on top of both a connectionless inter-network layer(IP)as well as over a connection-oriented internet protocol(Co-IP).We have conducted a systematic performance evaluation of the MCP kernel implementation and have also compared the performance of user applications with and without MCP. Our experience with MCP has revealed the utility of its abstractions in implementing collaborative software. It also demonstrates the performance gains achieved by designing a new protocol instead of implementing the MCP functionality at the application layer.The rest of this paper is organized as follows.Section2presents the background motivation for our research.In Section3,we describe the proposed communication abstractions.A protocol framework for implementation of the abstractions is proposed in Section4.Section5describes a prototype implementation and Section6presents the results of a performance evaluation.In Section7,we compare our work against other relevant research,andfinally,Section8concludes the paper.2Coordination and Control ProblemIn the design of multimedia collaborative systems,one must address three coordination and control issues:Concurrency Control:Multimedia collaboration typically involves one or more multipointflows1.Each flow involves several participants.Each participant may either be an application program(such as a design tool component)or a human user(e.g.a voice conferee).When several participants may concurrently transmit data over a sharedflow,the need for concurrency control arises to ensure consistent views and smooth interactions.Examples of such control include avoiding interruptions and crosstalk in a multipoint voice channel[Sch92,CLSF93],or maintaining consistency of a shared text document or an image[Dew90]in a collaborative application.As explained in Section2.1below,the desired degree of concurrency control varies over a wide range depending on the application or the context.Temporal Synchronization:A multimedia application needs to synchronize traffic over multipleflows, where eachflow carries traffic from a different medium(text,voice,video,or image).A collabo-rative,software development group effort described earlier(in section1.1)is an example of such an interaction.For such an application,the communication software must allow related streams to begrouped together,recognize the order and sequencing of messages sent over them,and preserve such ordering in delivering messages to each participant.For example,when a participant in a group editor scrolls through an image browser(or a shared editor window)and says,“look at the middle of the display”,the statement should be heard at the same time (or just after)the scrolling is completed.Such temporal relationships must be captured in the delivery of traffic over related streams.Another example of such a synchronization is“lip-synching”when voice and video are transmitted over separateflows in a network.Causal Synchronization:When collaboration involves multiple participants communicating either over a singleflow or using multipleflows,causal relationships among messages sent over theflow(s)must be maintained to preserve the context in which a message is sent.Consider the case of a voice conference where a singleflow is shared among multiple participants.When a participant B responds with message2after hearing message1from another participant A,a third participant C must receive and interpret messages1and2in the proper causal order to make perfect sense.Maintenance of such causality is especially important when interaction involves programs(rather than human users)updating a shared document or image.Moreover,in collaboration involving multiple media,such causal relationships exist among messages sent over separateflows.Consider a collaborative design editor involving twoflows:an imageflow to carry messages containing drawings and updates of shared images,and a voiceflow for conversation among participants.In such an interaction,if a participant scrolls through or modifies the shared image(resulting in a message update over the imageflow)in response to a voice message from another participant,then a causal relationship exists among messages sent over two differentflows.Again,as explained below,the desired degree of causality varies depending on the media involved and the application.2.1Degree of CoordinationIn the following,we describe how different applications and contexts require a varying degree of concurrency control and temporal/causal synchronization.Concurrency Control:The degree of concurrency control needed varies from one application to another.In addition,an application may go through several phases of interaction and each phase may need different degree of concurrency control.At one extreme is the case of a shared window package[SFB87,AEHL88]where a single window is displayed on the display screens of multiple users,and each user gets an identical image of the window.Many shared window systems provide no concurrency control for simultaneous actions, allow users to constantly see the actions of other users who are responsible for manually ensuring that there are no conflicts.At the other extreme lie teleconferencing systems where interference is avoided by strict control based on the notion of a“floor”,where only the current“speaker”that has thefloor is allowed to“speak”(or transmit data)at any time.Both the extremes have their limitations.On one hand,lack of any concurrency control puts additional responsibility on application designers to resolve conflicts.On the other hand,strictfloor-based control does not allow applications to exploit inherent concurrency and may sometimes unnecessarily increase latency due to the waiting involved.Needs of some applications may lie between the two extremes.For instance,users in a conferencing system may sometimes decide to enter smaller discussion groups that may separately hold concurrent conversations[SFB87].An interaction based on the“chalkboard”metaphor[SFB87]in a coop-erative environment is another example where less stringent coordination is appropriate.In the case of a group editor,strict coordination is not necessary when one user is browsing through a part of the shared text while another user is annotating a different part of the text.Thus,aflexible concurrency control mechanism is necessary to allow a wide range of semantics. Causal Synchronization:Problem of causal delivery has been addressed before in the case of distributed systems[BJ87,PBS89].When collaboration only involves shared text or still images,enforcement of strict causality is similar to the previous work in distributed systems because there are no real-time constraints on message delivery.However,multimedia collaboration involves delay sensitive media such as voice and video.V oice or video communication is delay sensitive in nature in that the data must be presented at specific instants(or at specific rate)at a receiver to convey its original meaning.The delivery of data units must follow better never than late semantics and multimedia messages that are delayed beyond an upper bound are rendered useless for playback.Thus,the problem of both causal and temporal synchronization is complicated when delivery of delay-sensitive multimedia data is involved.Within an application,desired level of causality may depend on the context and the media involved,and may vary from strict causality to less strict notions of causality.The communication software must provide mechanisms that allow an application to enforce the necessary semantics.3Communication AbstractionsGiven the need forflexible coordination and synchronization mechanisms,our design is based on the principle of separation of policy and mechanism.We believe that the right way for accommodating a wide range of coordination and synchronization requirements is to provide communication mechanisms that allow specification and selection of the degree of coordination and control needed and leave the task of choosing the appropriate policy to the application designer.We propose two methods of coordination:token-based control and an abstraction called multi-flow conversation.Together,these two methods yield aflexible and adaptable coordination mechanism as discussed below.3.1Token-based Concurrency ControlWhen a multipointflow is created,a token is assigned to thatflow that acts as an authorization for data transmission.A sender must hold a token to be able to send traffic over aflow.However,token management primitives are provided so that other participants can obtain transmission privileges.Thus,applications are free to transfer,replicate,and delete tokens to govern the degree of concurrency control needed.This type of concurrency control is entirely different from the token-based synchronization provided by some session-layer protocols.The latter method of synchronization allows session participants to insert resynchronization points(or checkpoints)in the data stream to allow rollback and to reduce the amount of retransmitted data in case of a transmission error.The token mechanism allows several levels of concurrency control.For example,we describe be-low a concurrency control hierarchy derived based on the collaboration paradigms used in existing sys-tems[SFB87,AEHL88,CF89,SG85].Floor Control Real-time teleconferencing systems employ such a strict concurrency control.A single token enforces control over a multipoint connection.The token will be passed on from one speaker to another whenever thefloor is transferred.To obtain control of thefloor,a participant must explicitly request transfer of the token.Chalkboard Interaction Applications based on the“chalkboard”metaphor in a cooperative environment interact in two phases.In thefirst phase,a speaker addresses a group of listeners with no interruptions.The application may switch to the second phase any time.In the second phase,a group of questioners may address questions to the speaker.The token-based method can accommodate both types of interaction.Only the speaker may hold the token during thefirst phase,whereas the token may be replicated and distributed to the questioners during the second phase.Replicated tokens will be destroyed at the end of the second phase.Discussion Groups Some environments such as real-time conferencing systems[CF89,SG85]envisage a session breaking up into smaller discussion groups and thus holding multiple,concurrent conversa-tions.In such a system,initially only a single token would be created and passed on from one speaker to another to achieve strict“floor-based”control.However,the token can be replicated and transferred to each discussion group,and each group may then use the token independently as it seesfit.Brainstorming This form of coordination is common in shared window systems where there is no concur-rency control for simultaneous actions by multiple users.For such applications,the token is replicated and distributed to all the participants.3.2Multi-flow ConversationsTo allow temporal and causal synchronization among traffic over multipleflows,we introduce a communi-cation abstraction called a multi-flow conversation.A conversation is a logical entity and consists of one or more(two-party or multipoint)flows.An application will typicallyfirst create individualflows with appropriate performance requirements[Fer90].It will then create a conversation that includes one or more relatedflows to achieve the necessary temporal and causal synchronization.For example,Figure1shows a conversation C that consists of twoflows,1and2.If a sender1sends a message1overConversationS1S2S3(voice)Flow1Flow2(video)Control Flow Figure 1:Conversation C consists of two flows1and 2that span three participants 1,2,and 3.2and then sends another message 2over 1,all the participants should receive those messages in the same temporal order.Moreover,if the participant 2sends a message3after receiving message 2,the participant 3must “see”the correct causal order (1followed by 2followed by 3).The conversation abstraction ensures such ordering.It uses message boundaries as synchronization markers.Messages sent over flows from the same sender are assigned monotonically increasing sequence numbers and are also timestamped to capture the temporal relationships among messages sent over different flows.In the following,we describe the details of temporal and causal synchronization.3.3Temporal and Causal RelationshipsFormally,let denote the set of flows in a conversation C and let denote the set of messages sent over the constituent flows in .Definition 1.We define (“precedes”)to be a transitive relation on M,such that 12if and only if the following conditions hold:1.both 1and 2are sent by the same sender,and2.both1and2are sent overflows and(both),and3.the message1is sent before2is sent.The sequence numbers assigned to messages sent over constituentflows by the same sender capture the partial order defined by the relation in MCP.In addition,timestamps assigned to each message precisely identify the temporal relationships among related messages.However,this temporal relationship must be augmented further to provide causally ordered delivery of messages originating from multiple senders.Stricter notions of causality provided by a causal broadcast[BJ87,PBS89]achieve a global ordering of messages necessary for this purpose.Following Lamport’s definition of causality[Lam78],we define a causal dependency relation“”among messages in a conversation as:Definition2.12if and only if at least one of the following two conditions hold:1.12,or2.2is sent over someflow by sender1after receiving1over someflow,and bothThus,Definition2defines a partial order among all the messages exchanged over a conversation that captures both temporal and causal relationships.In the following,we refer to such an order simply as causal order even though it includes both causal and temporal relationships.Enforcement of such a causal order in the presence of varying delays and losses across a wide area network can cause problems for multimedia information.Delivery of a message may sometimes have to be delayed indefinitely waiting for other messages that precede it in a causal order.Such delays are not acceptable to real-timeflows involving voice or video that have better-never-than-late semantics.For instance,some real-time voiceflows typically require delivery of each sample within100milliseconds to maintain the desired playback rate at a receiver.In such cases,an application can adapt to some increases in delays,but delays of300milliseconds or more render a voice sample useless for playback.To take such delivery constraints into account,we discard strictly causal ordering in favor of a more adaptive notion of causality called∆-causality.Informally,∆-causality works as follows.Given a set offlows and a set of participants in a conversation, message delivery to an application at a recipient site is both temporally and causally ordered provided anupper bound∆on end-to-end delay is not violated for any of the related messages.In the case of example involving Figure1,the receiver3will see the proper causal order provided1and2were generated and sent in such a way that their arrival at3is not delayed beyond an upper bound∆.∆is a function of individual delay constraints forflows1and2.To take delay constraints into account,we define a∆-Causal order as:Definition3.1∆2if:1.12and2.1is sent at most∆time units before2It must be noted that our notion of real-time assumes large-grain clock synchronization among the participants.This is not an unrealistic assumption as fault-tolerant clock synchronization algorithms exist that achieve such synchronization.In the TCP/IP Internet,Network Time Protocol(NTP)achieves global clock synchronization across the country within a few milliseconds[Mil89].Thus a small upper bound (typically5-10milliseconds)on difference between clocks at any two participants can be assumed.In the following,we describe∆-Causality in more detail.3.3.1∆-CausalityWe assume that the network layer provides aflow abstraction with performance guarantees to provide predictable service needed by real-time voice or video channels[Zha89,FV90,CY89].Desired performance characteristics are specified when aflow is established.Apart from the usual packet rate and error rate parameters(not considered here),there are two delay constraints associated with eachflow:Desired delay is the maximum end-to-end delay that theflow can tolerate before quality of ser-vice deteriorates.Example of such a constraint is100milliseconds delay bound in packet voice.Ferrari[Fer90]gives several examples of delay constraints for different classes of traffic.Loss delay is an upper limit on end-to-end delay forflow traffic beyond which the delivered traffic delivery is useless.For example,packet voice or video have better-never-than-late delivery semanticsand specify such a constraint.Packet voice traffic with desired delay constraint of100ms has a loss delay constraint in the range of200to300ms.Consider a conversation C consisting offlows1with respective delay constraints’s and ’s.We compute two conversation-wide desired delay and loss delay constraints:∆11∆21Based on these two constraints,we define the causality interval∆for each conversation.∆is computed as∆1∆∆2and defines a window of causality for each message sent in a conversation.For example, if a message4is sent by a participant after receiving messages1through3over some constituent flows,4must normally be received by applications at all other sites in the correct causal order(i.e.,after 1through3are received).However,∆-causality specifies that a conversation provider at each receiving site maintain such causal relationship between4and each of its predecessors only if messages related to 4are received within interval∆.The value of∆is chosen based on the following considerations.First,∆must at least be equal to∆12 (is the maximum possible clock skew).Second,to allow someflexibility in the presence offluctuations in network conditions and delays,MCP provider may choose value of∆to be higher than∆12without compromising individualflow semantics as long as∆remains much below the upper bound∆2.The interval∆is used as follows:1.Each conversation is assigned a network-wide unique conversation identifier conv_id;each messagein a conversation is timestamped(msg_time)and is also assigned a conversation-wide unique mesg_id2.2.Whenever a participant at a site1sends a message1over one of the constituentflows of aconversation,the conversation provider at that site includes some context for that message.3.The context information included with each message is a pair<sender_id,mesg_id>foreach participant.That is,the context includes the sequence number of the last message received at1from each participant before sending1.Thus,the amount of context is bounded by the number of conversation participants.The amount of context is also bounded by the causality interval∆,because not all the context information may be current and useful if last message from some participant was not received in recent past.In particular,causality interval implies that a message received earlier at1provides no context for1if was received at time less than∆if is the time at which1is sent by 1.4.When a receiver1at another site receives message1,it checks to see whether or not it has alreadyreceived all the messages specified in the context of1.If not,it buffers1waiting for the missing messages to be delivered and notes the timestamp1for1.It may have to buffer additional messages in1’s context if they also arrive out of order.However,if the missing messages are not received within the interval∆after1,the conversation provider delivers1to its application and deallocates buffers.The parameter∆essentially acts as a tuning knob;when∆is zero a conversation only guarantees causal synchronization among messages originating from the same sender and when∆goes to infinity you get strict causality similar to causal broadcast in ISIS[BJ87].The latter is useful when dealing with non real-time (“batch”)multimedia applications such as mail or a slide presentation playback where a remote server is sending data from a multimedia storage.Note that the idea of using and preserving context information is not completely new as it has been used before in distributed systems[PBS89,BJ87].However,we have extended it in two directions.First,we use the context information across traffic over multipleflows in a conversation rather than treating it separately within each individualflow.Second,the notion of causality provided is weaker and bound by real-time traffic delivery constraints as demanded by multimedia distributed applications.4Multi-Flow Conversation Protocol FrameworkTo realize the communication abstractions proposed earlier,we have designed a MCP protocol framework and Figure2shows the proposed communication architecture based on layering in the ARPA Internet protocol suite.At the network layer,we assume a reservation-based service such as the one provided by theSessionTransportMCP TCP UDPIP COIPToken ControlConversations Flow Control(or ST-II)Figure 2:Organization of the MCP protocol framework in the ARPA Internet hierarchy.Tenet Scheme Channels [FV90,FBZ92],ST-II [LT90],and RSVP [ZDE 93].However,our current design is based on McHIP and we assume that the network layer service guarantees an upper bound on end-to-end delay for each flow.Above the network layer,the MCP functionality is subdivided into transport and session layer mech-anisms.At the transport layer,MCP needs a flow control mechanism and can use an existing transport protocol such as UDP 3.At the session layer,MCP implements the conversation abstraction,multipoint flow management,and token management.A session layer MCP provider resides at each site and interacts with its peers residing at other sites to to coordinate management of flows and conversations.Applications interact with the MCP session layer provider through a set of service primitives.4.1Flow ManagementUnder flow management,MCP provides service primitives for flow creation,flow termination,and prim-itives for sending and receiving messages.An application establishes a flow by using the service call init_flow(flow name,params,list of participants).The init_flow returns a flow identifier (a flow_id )and a token for coordinating data transfer.The user-specified (that is agreed upon in advance)“flow name”(in addition to the flow identifier)serves。