Distributed MapReduce Engine with Fault Tolerance

Distributed MapReduce Engine with Fault ToleranceLixing Song,Shaoen Wu Dept.of Computer Science Ball State UniversityMuncie,IN {lsong,swu}@Honggang WangDept.of Electrical and Computer EngineeringUniversity of Massachusetts DartmouthDartmouth,MAhwang1@Qing YangComputer Science DepartmentMontana State UniversityBozeman,MTqing.yang@Abstract—Hadoop is the de facto engine that drives current cloud computing practice.Current Hadoop architecture suffers from single point of failure problems:its job management lacks of fault tolerance.If a job management fails,even if its tasks remains still active on cloud nodes,this job loses all state information and has to restart from scratch.In this work,we propose a distributed MapReduce engine for Hadoop with the Distributed Hash Table(DHT)algorithm that drives the scalable peer-to-peer networks today.The distributed Hadoop engine provides the fault-tolerance capability necessary to support efficient job computation required in the cloud computing with numerous jobs running at a moment.We have implemented the proposed distributed solution into Hadoop and evaluated its performance in job failures under various network deployments.I.I NTRODUCTIONCloud computing[1]–[4]is a model for enabling ubiq-uitous,convenient,on-demand network access to a shared pool of configurable computing resources(e.g.,networks, servers,storage,applications,and services)that can be rapidly provisioned and released with minimal management effort or service provider interaction.With the dramatic upsurging of cloud computing applications and business in the past years, Hadoop[5],[6],thanks to its open source implementation, has become the de facto engine enabling the current cloud computing.For example,large businesses such as Yahoo!, Facebook,Amazon and HP all adopt Hadoop as the foundation in their cloud computing service to billions of customers. Hadoop wasfirst developed by Yahoo!and then released as open source to the public.It has been evolving from itsfirst version into today’s the second version released in2012.One way that cloud computing is able to scale up and down easily is through a programming model called MapReduce. In Hadoop,the MapReduce model has a master node that receives user jobs,then breaks them into a number of tasks and delegates(maps)the tasks to other slave computing nodes. The master also assigns other nodes to merge the pieces of completed computations of the task from the mappers back into integrated pieces(and eventually into the completed job). These slave nodes are called reducers.MapReduce is highly 1)flexible in that it can work on large tasks that require large amounts of mappers and reducers just as easily as it can work on small tasks that only require very limited computing resources,2)scalable in that it can process a small number of user jobs as well as a huge number of user jobs by dispatching the actual computation onto thousands,even millions,of out-of-shelf computing nodes,3)fault-tolerant for tasks(but NOT jobs!!)in that if a mapper or reducer fails,then the master can still complete the job by re-delegating the task of the failed node to another node.Though,there exists one major potential point of failure in the current Hadoop MapReduce architecture that can result in significant performance degradation—the master.If the master node fails,its managed user jobs are not be able to be completed because there is currently no redundancy for the master node.To address the single point of failure(SPOF)issue on the master node of the MapReduce engine,this paper proposes a fault-tolerant architecture for the master node in MapReduce by employing the Distributed Hash Table algorithm[7]that is the core in the today’s widely used peer-to-peer networks e.g.BitTorrent[8],[9].The distributed peer-to-peer networked master nodes now allow for multiple masters to manage one MapReduce user job,or at least be aware that it exists. Therefore,in the event that one master node goes down, there will be at least one more master node still available and running for the same user job that can take over the job management duty for the failed master node.How a master node determines or locates its peer master nodes for the managed user jobs is enabled by the DHT algorithm.In the rest of the paper,Section II briefs the current Hadoop MapReduce engine architecture,discusses SPOF issue of this architecture and presents the motivations for this work.Then, the design and implementation of the proposed distributed MapReduce solution is detailed in Section III.Next,Section IV presents our evaluation of the performance of the proposed distributed architecture.The related work is reviewed in Sec-tion V.Finally,Section VI concludes this paper and discusses our future work.II.B ACKGROUND AND M OTIVATIONS When this work was performed,Hadoop MapReduce was still in itsfirst version(MapReduce1).Before this work was completed,Hadoop MapReduce had involved into its sec-ond version(MapReduce2).However,the research problem addressed in this work based on MapReduce1still exists in MapReduce2.Therefore,in this section,we review both versions since the implementation and evaluation were based on MapReduce1.A.Hadoop MapReduce Engine ArchitectureMapReduce1:Fig.1shows the framework architecture of MapReduce1.There are two key components in the architec-ture:master node(a.k.a jobtracker node)and slave nodes(a.k.a tasktracker nodes).A user job is decomposed(mapped)into a number of tasks by the MapReduce engine at the master nodeand those tasks are dispatched to the remote slave nodes that are close to the interesting data.The JobTracker managing the user job at the master node tracks the status and progress of all tasks of the job with the heartbeats from/to the TaskTrackers at the slave nodes that manages the tasks.After all tasks are finished,the JobTracker condenses(reduces)the returned task results into afinal result and returns to the user.Fig.1.MapReduce1ArchitectureMapReduce2:MapReduce1was upgaded to MapReduce2 (a.k.a YARN–Yet Another Resource Negotiator[10])to mainly address a scalability issue.In MapReduce1,the JobTracker at the master node actually takes both responsibilities:a)job scheduling—-identifying proper remote slave nodes for the mapped tasks,and b)tracking the progress and status of tasks of a managed user job.MapReduce1has a scalability bottleneck of about4,000nodes and is difficult to support very large clusters that are required in many business cases today. MapReduce2splits these two responsibilities to support very large scales.Fig.2shows the framework architecture of MapReduce2. The ResourceManager at the resource manager node takes care of the job scheduling by identifying the nodes to run the mapped tasks of a user job.The ApplicationMaster at themaster node tracks the progress and status ofthe tasks runningon slave nodes.The ApplicationMaster processes the returned task results and updates the user.Fig.2.MapReduce2Architecture B.Research Problems and MotivationsLet us inspect the consequence of failures in the Hadoop MapReduce architecture.As we have mentioned earlier,if a task fails at a salve node,both MapReduce1and MapReduce2 can take care of this failure by initiating another task,possibly at another node.However,if the master physical node crashes (less likely),or the JobTracker/ApplicationManager fails(more likely),the status of a running job at the master node is gone, even its tasks are still running on slave nodes.As a result, the user loses the track of his/her job.In MapReduce1,the job has to be restarted from scratch.In MapReduce2,the ResourceManager starts another instance of the user job,also from scratch.As we can see,if a user job requires extensive computation e.g.climate simulation,retuning the job takes a significant amount of resources and time.In MapReduce2, the problem becomes even worse if the ResouraceManage fails because no more user requests can can be accepted and the management of the cluster resources is out of control. Therefore,the Hadoop MapReduce Engine has a SPOF problem.It has been revealed that“a single failure can lead to large,variable and unpredictable job running times”[11], [12].This project is thereof motivated to address this problem and to provide fault tolerance for job management.The goal is that a job is NOT required to restart when the job management fails at one node.III.D ISTRIBUTED M AP R EDUCE E NGINE To address the SPOF problem on the user job management in the MapReduce architecture,we propose a distributed MapReduce engine that offers the fault tolerance on managing user jobs.A.Conceptual ArchitectureThe distributed MapReduce engine consists of a group of physically distributed nodes that collectively serve together as a master network as shown in Fig.3.A user job will be managed by more than one nodes in this master network.The master nodes of a user job synchronizes their images of the job.But at one time,only one master node serves as the active master node for the ly,the tasks of the job only communicates to one mater node—-the active master node. When the active master node,or its JobManager,fails,the standby master node(s)and the slave nodes can detect the failure with the heartbeats.Then,the new active master node will be elected from the standby master nodes upon a policy discussed later.The communication about the status of the tasks at the slave nodes will be directed to the new active master node so that the job continue its running.To the system and the user,it seems nothing happened in the job processing. The next discusses how the master network is formed and how the switch takes place when an active master node or its JobManager fails.B.Formation of Master Network with DHTThe key component in the proposed architecture is a dis-tributed master network.This network could be implemented with a separated physical network of nodes.For resource efficiency,we rather propose it be implemented as a virtual network.When a regular cloud node runs the job managementFig.3.Distributed MapReduce Conceptual Architecturefor a user job,this node is part of the master network and be referred as a mater node.It can also actually run regular computation task as a slave node for other ly,a physical node can serve as both a master node for certain jobs and a slave node for other jobs.As a result,the whole cloud could be a virtual master network at the extremity.It should be noted that the existence of the master node network for a job relies on the job existence—it is purged as a job is completed.In this proposed architecture,the master network is formed with DHT algorithm.First,a small number of cloud nodes are specified as server nodes in the DHT framework.When a user job arrives at a job management,the job management node acts as the active master node for the job and identifies its peer master nodes through the server nodes with the DHT algorithm as in Chord[7].The number of master nodes can be specified by the system(we recommend2to4nodes as the master nodes for a job since the physical node has a very low down rate).Because a network problem cloud result in the failure of all its nodes,a policy in identifying the peer master nodes is that a peer node should at best effort be in a deferent LAN for better fault-tolerance.Then,each master node of a user job will have a list of its peers in the order of their node IDs on the DHT ring.Then,the job management initiates the mapped tasks onto remote slave nodes.The task updates are performed as in cur-rent practice between the tasks and the their job management. Meanwhile,the active master node synchronizes the job status information among its standby peer master nodes,which is discussed next.C.Job Image SynchronizationTo guarantee the job continuity in the event of failure of the active master node,the master nodes of a job must have the same job images.There are two ways to achieve this.One is to have the tasks of a job to multicast their status and progress to all master nodes,but incurs expensive communication cost in the network.The other approach,which is adopted by us,is to have the tasks communicate their active master node only, but this requires the synchronization of job images among the master nodes.We propose an Incremental synchronization scheme with light cache.The active master node only sends the incremental update that is the difference from the last update to its peer master nodes.To guarantee the reception of the updates,the active master has to be acknowledged by the peer masters. Therefore,the synchronization is carried on TCP in unicast because TCP ensures the delivery and unicast allows ac-knowledgement(on the contrary,multicast does not support acknowledgement).Since the active master may fail at any moment,it is possible that some task progress received after the last synchronization but before the failure is lost in the failure.To address this issue,the task nodes have to keep their task progress for the time of a synchronization cycle.So,when the active node fails,the new active node solicits the missed task updates from the task nodes to keep up-to-date.D.Active Master SwitchingWhen the master node network of a job is formed as discussed in Section III-B,the master nodes are compiled into a list in ascending order according to their node IDs.This list is disseminated to all master nodes and tasks nodes so that every node knows which node is the next active master is the current active master ly,the next active master node is pre-elected based on the node IDs.If the standby master nodes do not receive the periodic updates as scheduled from their active peer master,they will send a status inquiry heartbeat to the active master.If no response of the inquiry is received,they consider the active master fails and the next master node in the master list will take over the active master role.All other master nodes will synchronize to this new active master.E.ImplementationImplementation on MapReduce1:We have implemented the proposed distributed MapReduce engine on Hadoop code base.The implementation was based on MapReduce1be-cause MapReduce2was not released when our implementa-tion started.The DHT algorithm was implemented into the JobTracker as in Fig. 1.The implementation of DHT is based on the source code of Open Chord[13].The mas-ter synchronization capability was also implemented in the JobTracker.Because of the MapReduce1architecture,in our implementation,the master nodes are a group of specific nodes that doe not run regular tasks as MapReduce2does.To accommodate the distributed JobTracker nodes,TaskTracker is changed to enable the short-term cache of updates and keep a list of the master nodes.Possible Implementation on MapReduce2:It should be noted that the implementation of the proposed distributed MapReduce solution onto MapReduce2is still feasible,but in a”flatter”mode.As in Fig.2,MapReduce2retains the ResourceManager as a separated node,but the AppManager has been dispatched into a regular cloud node as its tasks do.Therefore,the implementation of the DHT should be on the ResourceManager that is responsible for locating the nodes for AppManagers because the ResourceManager should generate and dispatch multiple AppManagers,rather than only one as it does without fault-tolerance currently.However,the synchronization should be implemented on the AppManagers because they need to communicate to each other.The taskJVM should implement the short-term cache and the list of masternodes as the TaskTracker does in MapReduce1.Moreover,the ResourceManager itself should have a distributed architecture too because it has a SPOF as well.The distributed Resource-Manager architecture could be accompllished with DHT as in the distributed JobTracker implementation in the MapReduce1.IV.E VALUATIONWe have evaluated the distributed MapReduce solution implemented on MapReduce1.The evaluation focus is on the latency,success ratio of the master switch in failure cases, and the incurred network traffic overhead.Wefirst present the evaluation platform and methodologies,then discuss the evaluated metrics and the experiment results.A.Platform and MethodologiesThe evaluation was carried out on a Hadoop cloud con-sisting of virtual machines.We have two Ubuntu-Linux host machines configured into two LANs and each of them has five virtual machines installed and configured with the Hadoop cloud computing platform.The Hadoop platform includes both the MapReduce1package the Hadoop Distributed File System (HDFS).The evaluation configuration is illustrated in Fig.4.We set the master synchronization cycle as the ten times of the task update cycle.The short-term cache duration on the task nodes was then linked to the synchronization cycle in ly,the task nodes cached the last ten updates locally.The two virtual networks of the virtual machines were configured with network masks of192.168.0.255and 192.168.1.255respectively.The IP address of each node was used as the node ID for the DHT.Because of the limitation of the physical computing resources,only80jobs were submitted to run in the cloud.We configured each master network to contain only two master nodes(i.e.one active and the other standby).With the preference that master nodes should be separated in different networks if possible,the active master nodes were basically in the network of192.168.0.0and the standby master nodes were in192.168.1.0.We emulated the active master failure by purging the instance of a JobTracker of an active master node.Fig.4.Experiment Platform of HadoopB.Metrics and ResultsOur evaluation focused on three metrics:switch latency, switch success ratio and network overhead.The evaluation results of the three metrics are shown in Table I.Switch latency:This metric refers to the time from a failure of an active master to a new active master takes over the job management with its tasks.To avoid the error incurred by the system time difference between two host machines,we rather limited this experiment into only one network,namely both the active and the standby master nodes are in the same network on the same host machine.We measured1000master switches and averaged the latency.The latency was normalized by the synchronization cycle.From the measurements in Table I,we observe that the switch latency is a little over half a synchronization cycle.This is reasonable because the worst case occurs when a failure happens immediately after a synchronization and the standby node has to wait till the end of almost the whole cycle to detect the failure.Therefore the maximum latency should be of a synchronization cycle plus the inquiry message timeout.The minimum latency should be close to the timeout of inquiry message when a failure occurs nearly at the end of a synchronization cycle.Therefore, with the a uniform distribution of failure,the expected latency should be the half of the synchronization cycle plus the inquiry message timeout.Switch success ratio:this metric is defined as the number of successful master switches divided by the number of the switch attempts(or the number of active master failures).This metric is important in that it shows how much fault-tolerance is provided by the distributed MapReduce architecture.Ideally, the ratio should be100%,but it is hurt by network transmission such as packet loss.With the1000master switches tested,the measurement shows a success ratio of97%,which indicates that the distributed solution is effective in providing fault-tolerance to user jobs.Network overhead:This metric measures the overhead of network traffic incurred by formation of the master network, synchronization and active master switch.It should grow along with the size of the master network because more copies of a message has to be sent in synchronization and master switch. Therefore,the measurement result is normalized by the number of master switches.As observed from Table I,the traffic is about5messages per master switch in our case of two master nodes only and three tasks associated with a job.With each message takes nearly0.5μs in Gbps networks,the network cost is about2.5μs resulted from the switch in the distributed MapReduce solution.TABLE I.E VALUATION R ESULTSMetric ResultsLatency0.51Success Ratio0.97Network Overhead0.5μsV.R ELATED W ORKHadoop is essentially an open source massively scalable queryable store and archive platform enabling the cloud com-puting that includes afile system,queryable databases,archival store,andflexible schema[14].Hadoop can,and normally does,use the Hadoop Distributed File System(HDFS)that uses the write-once,read-many philosophy meaning that instead of the hard disk constantly having to seek to modify data throughout the set,any new data is appended to the end of the current dataset.MapReduce is the programming model thatwas developed by Google[15]and that Hadoop uses to process data[6].The datasets processed in Hadoop can be,and often are,much larger than any one computer can ever process[16]. MapReduce organizes how that data is processed over many computers(anywhere from a few to a few thousand)[16].Distributed Hash Table(DHT)wasfirst proposed in the work of Chord[7]by MIT to support overlay peer-to-peer network for contention distribution with some other DHT implementations available such as CAN[17],Pastry[18], and Tapestry[19].Chord is a distributed lookup protocol that specializes in mapping keys to nodes[8].Data location can be easily implemented on top of Chord by associating a key with each data item,and storing the key/data pair at the node to which the key maps.Chord adapts efficiently as nodes join and leave the system,and can answer queries even if the system is continuously changing.This is especially a goodfit to MapReduce because MapReduce already maps key-value pairs to process data,and mappers and reducers are constantly being brought up and shutdown[16].Chord features load balancing because it acts as a distributed hash function and spreads keys evenly over the participating nodes.Chord is also decentralized;no node is of greater importance than any other node.Chord scales automatically without need to do any tuning to achieve success at scale.Some other fault-tolerance efforts have been proposed for cloud computing.Cassandra[20]is a peer-to-peer based solu-tion proposed by Facebook engineers to address fault-tolerance in distributed databased management.It eliminates the SPOF problem in a distributed data storage.Other effort in addressing data fault-tolerance includes work like[21].YARN[10]is proposed in the MapReduce2of Hadoop to address the fault-tolerance in HDFS.It solves the SPOF problem in an HDFS cluster.So far,there has been no solution proposed to address the SPOF problem on the job management as our this work does.VI.A CKNOWLEDGEMENTThis paper presents a fault-tolerant MapReduce engine for cloud computing.The fault-tolerance is enabled by a distributed solution based on DHT algorithm.In the solution, a network of master nodes are formed to provide job man-agement.A failed active master node will be replaced by its next standby peer node for job management and thereby the job running is maintained.The solution has been implemented into the Hadoop MapReduce1engine and evaluated of high fault-tolerance with low latency and networking cost.Our next step is to implement this solution into the current Hadoop MapReduce2for more extensive performance evaluation and release it as open source.VII.A CKNOWLEDGEMENTThe authors would like to thank Gordon Pettey who helped with the extensive source code implementation of the solution.A special thanks is given to National Science Foundation for the award#1041292that supports Gordon to work on this project as a REU student.We would like also thank our reviewers for their precious comments to make the work better.R EFERENCES[1]M.Armbrust,A.Fox,R.Griffith,A.D.Joseph,R.Katz,A.Konwinski,G.Lee,D.Patterson,A.Rabkin,I.Stoica,and M.Zaharia,“A viewof cloud computing,”Commun.ACM,vol.53,no.4,pp.50–58,Apr.2010.[Online].Available:/10.1145/1721654.1721672 [2]P.Mell and T.Grance,“The nist definition of cloud computing(draft),”NIST special publication,vol.800,no.145,p.7,2011.[3]T.Velte,A.Velte,and R.Elsenpeter,Cloud computing,a practicalapproach.McGraw-Hill,Inc.,2009.[4]Q.Zhang,L.Cheng,and R.Boutaba,“Cloud computing:state-of-the-artand research challenges,”Journal of Internet Services and Applications, vol.1,no.1,pp.7–18,2010.[5] D.Borthakur,“The hadoop distributedfile system:Architecture anddesign,”2007.[6]T.White,Hadoop:the definitive guide.O’Reilly,2012.[7]I.Stoica,R.Morris,D.Liben-Nowell,D.R.Karger,M.F.Kaashoek,F.Dabek,and H.Balakrishnan,“Chord:a scalable peer-to-peer lookupprotocol for internet applications,”IEEE/ACM w.,vol.11, no.1,pp.17–32,2003.[8] B.Cohen,“The bittorrent protocol specification,”2008.[9] D.Qiu and R.Srikant,“Modeling and performance analysis ofbittorrent-like peer-to-peer networks,”ACM SIGCOMM Computer Communication Review,vol.34,no.4,pp.367–378,2004.[10] A. C.Murthy, C.Douglas,M.Konar,O.O’MALLEY,S.Radia,S.Agarwal,and V.KV,“Architecture of next generation apache hadoop mapreduce framework,”Tech.rep.,Apache Hadoop,Tech.Rep.,2011.[11] F.Dinu and T.Ng,“Understanding the effects and implications ofcompute node related failures in hadoop,”in Proceedings of the21st international symposium on High-Performance Parallel and Distributed Computing.ACM,2012,pp.187–198.[12] F.D.T.E.Ng,“Analysis of hadoop’s performance under failures,”RiceUniversity,Tech.Rep.,2012.[13]L.Karsten and K.Sven,“.”[14] D.Borthakur,J.Gray,J.S.Sarma,K.Muthukkaruppan,N.Spiegelberg,H.Kuang,K.Ranganathan,D.Molkov,A.Menon,S.Rash et al.,“Apache hadoop goes realtime at facebook,”in Proceedings of the 2011ACM SIGMOD International Conference on Management of data.ACM,2011,pp.1071–1080.[15]J.Dean and S.Ghemawat,“Mapreduce:simplified data processing onlarge clusters,”Communications of the ACM,vol.51,no.1,pp.107–113,2008.[16]J.Lin and C.Dyer,“Data-intensive text processing with mapreduce,”Synthesis Lectures on Human Language Technologies,vol.3,no.1,pp.1–177,2010.[17]S.Ratnasamy,P.Francis,M.Handley,R.Karp,and S.Shenker,Ascalable content-addressable network.ACM,2001,vol.31,no.4. [18] A.Rowstron and P.Druschel,“Pastry:Scalable,decentralized object lo-cation,and routing for large-scale peer-to-peer systems,”in Middleware 2001.Springer,2001,pp.329–350.[19] B.Y.Zhao,L.Huang,J.Stribling,S.C.Rhea,A.D.Joseph,andJ.D.Kubiatowicz,“Tapestry:A resilient global-scale overlay for service deployment,”Selected Areas in Communications,IEEE Journal on, vol.22,no.1,pp.41–53,2004.[20] kshman and P.Malik,“Cassandra:a decentralized structuredstorage system,”ACM SIGOPS Operating Systems Review,vol.44, no.2,pp.35–40,2010.[21]S.Y.Ko,I.Hoque,B.Cho,and I.Gupta,“Making cloud intermediatedata fault-tolerant,”in Proceedings of the1st ACM symposium on Cloud computing.ACM,2010,pp.181–192.。