A Scalable Content-Addressable Network

A Scalable Content-Addressable Network Sylvia Ratnasamy Paul Francis Mark Handley Richard KarpScott ShenkerDept.of Electrical Eng.&Comp.Sci.ACIRIUniversity of California,Berkeley AT&T Center for Internet Research at ICSIBerkeley,CA,USA Berkeley,CA,USAABSTRACTHash tables–which map“keys”onto“values”–are an essential building block in modern software systems.We believe a similar functionality would be equally valuable to large distributed systems.In this paper,we intro-duce the concept of a Content-Addressable Network(CAN)as a distributed infrastructure that provides hash table-like functionality on Internet-like scales.The CAN is scalable,fault-tolerant and completely self-organizing, and we demonstrate its scalability,robustness and low-latency properties through simulation.1.INTRODUCTIONA hash table is a data structure that efficiently maps“keys”onto “values”and serves as a core building block in the implementa-tion of software systems.We conjecture that many large-scale dis-tributed systems could likewise benefit from hash table functional-ity.We use the term Content-Addressable Network(CAN)to de-scribe such a distributed,Internet-scale,hash table.Perhaps the best example of current Internet systems that could potentially be improved by a CAN are the recently introduced peer-to-peerfile sharing systems such as Napster[14]and Gnutella[6]. In these systems,files are stored at the end user machines(peers) rather than at a central server and,as opposed to the traditional client-server model,files are transferred directly between peers. These peer-to-peer systems have become quite popular.Napster was introduced in mid-1999and,as of December2000,the soft-ware has been down-loaded by50million users,making it the fastest growing application on the Web.Newfile sharing systems such as Scour,FreeNet,Ohaha,Jungle Monkey,and MojoNation have all been introduced within the last year.While there remains some(quite justified)skepticism about the business potential of thesefile sharing systems,we believe their rapid and wide-spread deployment suggests that there are impor-tant advantages to peer-to-peer systems.Peer-to-peer designs har-ness huge amounts of resources-the content advertised through Napster has been observed to exceed7TB of storage on a single day,1without requiring centralized planning or huge investments in Private communication with Yin Zhang and Vern Paxson Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on thefirst page.To copy otherwise,to republish,to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.SIGCOMM’01,August27-31,2001,San Diego,California,USA.. Copyright2001ACM1-58113-411-8/01/0008...$5.00.hardware,bandwidth,or rack space.As such,peer-to-peerfile shar-ing may lead to new content distribution models for applications such as software distribution,file sharing,and static web content delivery.Unfortunately,most of the current peer-to-peer designs are not scalable.For example,in Napster a central server stores the in-dex of all thefiles available within the Napster user community. To retrieve afile,a user queries this central server using the de-siredfile’s well known name and obtains the IP address of a user machine storing the requestedfile.Thefile is then down-loaded di-rectly from this user machine.Thus,although Napster uses a peer-to-peer communication model for the actualfile transfer,the pro-cess of locating afile is still very much centralized.This makes it both expensive(to scale the central directory)and vulnerable(since there is a single point of failure).Gnutella goes a step further and de-centralizes thefile location process as ers in a Gnutella network self-organize into an application-level mesh on which re-quests for afile areflooded with a certain scope.Flooding on every request is clearly not scalable[7]and,because theflooding has to be curtailed at some point,may fail tofind content that is actu-ally in the system.We started our investigation with the question: could one make a scalable peer-to-peerfile distribution system?We soon recognized that central to any peer-to-peer system is the in-dexing scheme used to mapfile names(whether well known or dis-covered through some external mechanism)to their location in the system.That is,the peer-to-peerfile transfer process is inherently scalable,but the hard part isfinding the peer from whom to retrieve thefile.Thus,a scalable peer-to-peer system requires,at the very least,a scalable indexing mechanism.We call such indexing sys-tems Content-Addressable Networks and,in this paper,propose a particular CAN design.However,the applicability of CANs is not limited to peer-to-peer systems.CANs could also be used in large scale storage management systems such as OceanStore[10],Farsite[1],and Publius[13].These systems all require efficient insertion and re-trieval of content in a large distributed storage infrastructure;a scal-able indexing mechanism is an essential component of such an in-frastructure.In fact,as we discuss in Section5,the OceanStore system already includes a CAN in its core design(although the OceanStore CAN,based on Plaxton’s algorithm[15],is somewhat different from what we propose here).Another potential application for CANs is in the construction of wide-area name resolution services that(unlike the DNS)decou-ple the naming scheme from the name resolution process thereby enabling arbitrary,location-independent naming schemes.Our interest in CANs is based on the belief that a hash table-like abstraction would give Internet system developers a powerful design tool that could enable new applications and communicationmodels.However,in this paper our focus is not on the use of CANs but on their design.In[17],we describe,in some detail,one possi-ble application,which we call a“grass-roots”content distribution system,that leverages our CAN work.As we have said,CANs resemble a hash table;the basic oper-ations performed on a CAN are the insertion,lookup and deletion of(key,value)pairs.In our design,the CAN is composed of many individual nodes.Each CAN node stores a chunk(called a zone)of the entire hash table.In addition,a node holds information about a small number of“adjacent”zones in the table.Requests(insert, lookup,or delete)for a particular key are routed by intermediate CAN nodes towards the CAN node whose zone contains that key. Our CAN design is completely distributed(it requires no form of centralized control,coordination or configuration),scalable(nodes maintain only a small amount of control state that is independent of the number of nodes in the system),and fault-tolerant(nodes can route around failures).Unlike systems such as the DNS or IP routing,our design does not impose any form of rigid hierarchical naming structure to achieve scalability.Finally,our design can be implemented entirely at the application level.In what follows,we describe our basic design for a CAN in Sec-tion2,describe and evaluate this design in more detail in Section3 and discuss our results in Section4.We discuss related work in Section5and directions for future work in Section6.2.DESIGNFirst we describe our Content Addressable Network in its most basic form;in Section3we present additional design features that greatly improve performance and robustness.Our design centers around a virtual-dimensional Cartesian co-ordinate space on a-torus.2This coordinate space is completely logical and bears no relation to any physical coordinate system.At any point in time,the entire coordinate space is dynamically par-titioned among all the nodes in the system such that every node “owns”its individual,distinct zone within the overall space.For example,Figure1shows a2-dimensional coordinate space partitioned between5CAN nodes.This virtual coordinate space is used to store(key,value)pairs as follows:to store a pair(,),key is deterministically mapped onto a point in the coordinate space using a uniform hash function.The corresponding(key,value)pair is then stored at the node that owns the zone within which the point lies.To retrieve an entry corresponding to key,any node can apply the same deterministic hash function to map onto point and then retrieve the corresponding value from the point.If the pointis not owned by the requesting node or its immediate neighbors, the request must be routed through the CAN infrastructure until it reaches the node in whose zone lies.Efficient routing is therefore a critical aspect of a CAN.Nodes in the CAN self-organize into an overlay network that rep-resents this virtual coordinate space.A node learns and maintains the IP addresses of those nodes that hold coordinate zones adjoin-ing its own zone.This set of immediate neighbors in the coordinate space serves as a coordinate routing table that enables routing be-tween arbitrary points in this space.We will describe the three most basic pieces of our design:CAN routing,construction of the CAN coordinate overlay,and mainte-nance of the CAN overlay.2.1Routing in a CANFor simplicity,the illustrations in this paper do not show a torus, so the reader must remember that the coordinate space wraps.Intuitively,routing in a Content Addressable Network works by following the straight line path through the Cartesian space from source to destination coordinates.A CAN node maintains a coordinate routing table that holds the IP address and virtual coordinate zone of each of its immediate neighbors in the coordinate space.In a-dimensional coordinate space,two nodes are neighbors if their coordinate spans overlap along dimensions and abut along one dimension.For example, in Figure2,node5is a neighbor of node1because its coordinate zone overlaps with1’s along the Y axis and abuts along the X-axis. On the other hand,node6is not a neighbor of1because their co-ordinate zones abut along both the X and Y axes.This purely local neighbor state is sufficient to route between two arbitrary points in the space:A CAN message includes the destination coordinates. Using its neighbor coordinate set,a node routes a message towards its destination by simple greedy forwarding to the neighbor with coordinates closest to the destination coordinates.Figure2shows a sample routing path.For a dimensional space partitioned into equal zones,the av-erage routing path length is hops and individual nodes maintain neighbors3.These scaling results mean that for a-dimensional space,we can grow the number of nodes(and hence zones)without increasing per node state while the average path length grows as.Note that many different paths exist between two points in the space and so,even if one or more of a node’s neighbors were to crash,a node can automatically route along the next best available path.If however,a node loses all its neighbors in a certain direction, and the repair mechanisms described in Section2.3have not yet rebuilt the void in the coordinate space,then greedy forwarding may temporarily fail.In this case,a node may use an expanding ring search(using stateless,controlledflooding over the unicast CAN overlay mesh)to locate a node that is closer to the destination than itself.The message is then forwarded to this closer node,from which greedy forwarding is resumed.2.2CAN constructionAs described above,the entire CAN space is divided amongst the nodes currently in the system.To allow the CAN to grow in-crementally,a new node that joins the system must be allocated its own portion of the coordinate space.This is done by an existing node splitting its allocated zone in half,retaining half and handing the other half to the new node.The process takes three steps:1.First the new node mustfind a node already in the CAN.2.Next,using the CAN routing mechanisms,it mustfind a nodewhose zone will be split.3.Finally,the neighbors of the split zone must be notified sothat routing can include the new node.BootstrapA new CAN nodefirst discovers the IP address of any node cur-rently in the system.The functioning of a CAN does not depend Recently proposed routing algorithms for location services[15, 20]route in hops with each node maintaining neighbors.Notice that were we to select the number of dimensions d=,we could achieve the same scaling properties.We choose to hold dfixed independent of n,since we envision apply-ing CANs to very large systems with frequent topology changes. In such systems,it is important to keep the number of neighbors independent of the system sizeFigure 1:Example 2-d space with 5nodes7’s coordinate neighbor set = { }152364(x,y)sample routing path from node 1 to point (x,y)1’s coordinate neighbor set = {2,3,4,5} Figure 2:Example 2-d space before node 7joins7523461’s coordinate neighbor set = {2,3,4,7} 7’s coordinate neighbor set = {1,2,4,5}1Figure 3:Example 2-d space after node 7joinson the details of how this is done,but we use the same bootstrap mechanism as YOID [4].As in [4]we assume that a CAN has an associated DNS domain name,and that this resolves to the IP address of one or more CAN bootstrap nodes.A bootstrap node maintains a partial list of CAN nodes it believes are currently in the system.Simple techniques to keep this list reasonably current are described in [4].To join a CAN,a new node looks up the CAN domain name in DNS to retrieve a bootstrap node’s IP address.The bootstrap node then supplies the IP addresses of several randomly chosen nodes currently in the system.Finding a ZoneThe new node then randomly chooses a point in the space and sends a JOIN request destined for point .This message is sent into the CAN via any existing CAN node.Each CAN node then uses the CAN routing mechanism to forward the message,until it reaches the node in whose zone lies.This current occupant node then splits its zone in half and assigns one half to the new node.The split is done by assuming a certain ordering of the dimensions in deciding along which dimension a zone is to be split,so that zones can be re-merged when nodes leave.For a 2-d space a zone would first be split along the X dimension,then the Y and so on.The (key,value)pairs from the half zone to be handed over are also transfered to the new node.Joining the RoutingHaving obtained its zone,the new node learns the IP addresses of its coordinate neighbor set from the previous occupant.This set is a subset of the previous occupant’s neighbors,plus that occupant itself.Similarly,the previous occupant updates its neighbor set to eliminate those nodes that are no longer neighbors.Finally,both the new and old nodes’neighbors must be informed of this realloca-tion of space.Every node in the system sends an immediate update message,followed by periodic refreshes,with its currently assigned zone to all its neighbors.These soft-state style updates ensure that all of their neighbors will quickly learn about the change and will update their own neighbor sets accordingly.Figures 2and 3show an example of a new node (node 7)joining a 2-dimensional CAN.The addition of a new node affects only a small number of ex-isting nodes in a very small locality of the coordinate space.The number of neighbors a node maintains depends only on the dimen-sionality of the coordinate space and is independent of the totalnumber of nodes in the system.Thus,node insertion affects only O(number of dimensions)existing nodes,which is important for CANs with huge numbers of nodes.2.3Node departure,recovery and CAN main-tenanceWhen nodes leave a CAN,we need to ensure that the zones they occupied are taken over by the remaining nodes.The normal pro-cedure for doing this is for a node to explicitly hand over its zone and the associated (key,value)database to one of its neighbors.If the zone of one of the neighbors can be merged with the departing node’s zone to produce a valid single zone,then this is done.If not,then the zone is handed to the neighbor whose current zone is smallest,and that node will then temporarily handle both zones.The CAN also needs to be robust to node or network failures,where one or more nodes simply become unreachable.This is han-dled through an immediate takeover algorithm that ensures one of the failed node’s neighbors takes over the zone.However in this case the (key,value)pairs held by the departing node are lost until the state is refreshed by the holders of the data 4.Under normal conditions a node sends periodic update messages to each of its neighbors giving its zone coordinates and a list of its neighbors and their zone coordinates.The prolonged absence of an update message from a neighbor signals its failure.Once a node has decided that its neighbor has died it initiates the takeover mechanism and starts a takeover timer running.Each neighbor of the failed node will do this independently,with the timer initialized in proportion to the volume of the node’s own zone.When the timer expires,a node sends a TAKEOVER message conveying its own zone volume to all of the failed node’s neighbors.On receipt of a TAKEOVER message,a node cancels its own timer if the zone volume in the message is smaller that its own zone volume,or it replies with its own TAKEOVER message.In this way,a neighboring node is efficiently chosen that is still alive and has a small zone volume 5.Under certain failure scenarios involving the simultaneous fail-ure of multiple adjacent nodes,it is possible that a node detects To prevent stale entries as well as to refresh lost entries,nodes that insert (key,value)pairs into the CAN periodically refresh these entriesAdditional metrics such as load or the quality of connectivity can also be taken into account,but in the interests of simplicity we won’t discuss these further here.a failure,but less than half of the failed node’s neighbors are still reachable.If the node takes over another zone under these circum-stances,it is possible for the CAN state to become inconsistent.Insuch cases,prior to triggering the repair mechanism,the node per-forms an expanding ring search for any nodes residing beyond thefailure region and hence it eventually rebuilds sufficient neighborstate to initiate a takeover safely.Finally,both the normal leaving procedure and the immediatetakeover algorithm can result in a node holding more than onezone.To prevent repeated further fragmentation of the space,abackground zone-reassignment algorithm,which we describe inAppendix A,runs to ensure that the CAN tends back towards onezone per node.3.DESIGN IMPROVEMENTSOur basic CAN algorithm as described in the previous sectionprovides a balance between low per-node state(for a -dimensional space)and short path lengths with hops for dimensions and nodes.This bound applies to the numberof hops in the CAN path.These are application level hops,not IP-level hops,and the latency of each hop might be substantial;recall that nodes that are adjacent in the CAN might be many miles and many IP hops away from each other.The average total latency of a lookup is the average number of CAN hops times the average la-tency of each CAN hop.We would like to achieve a lookup latency that is comparable within a small factor to the underlying IP path latencies between the requester and the CAN node holding the key. In this section,we describe a number of design techniques whose primary goal is to reduce the latency of CAN routing.Not unin-tentionally,many of these techniques offer the additional advan-tage of improved CAN robustness both in terms of routing and data availability.In a nutshell,our strategy in attempting to reduce path latency is to reduce either the path length or the per-CAN-hop la-tency.Afinal improvement we make to our basic design is to add simple load balancing mechanisms(described in Sections3.7and 3.8).First,we describe and evaluate each design feature individually and then,in Section4,discuss how together they affect the overall performance.These added features yield significant improvements but come at the cost of increased per-node state(although per-node state still remains independent of the number of nodes in the sys-tem)and somewhat increased complexity.The extent to which the following techniques are applied(if at all)involves a trade-off be-tween improved routing performance and system robustness on the one hand and increased per-node state and system complexity on the other.Until we have greater deployment experience,and know the application requirements better,we are not prepared to decide on these tradeoffs.We simulated our CAN design on Transit-Stub(TS)topologiesusing the GT-ITM topology generator[22].TS topologies modelnetworks using a2-level hierarchy of routing domains with transitdomains that interconnect lower level stub domains.3.1Multi-dimensioned coordinate spacesThefirst observation is that our design does not restrict the di-mensionality of the coordinate space.Increasing the dimensions of the CAN coordinate space reduces the routing path length,and hence the path latency,for a small increase in the size of the coor-dinate routing table.Figure4measures this effect of increasing dimensions on rout-ing path length.We plot the path length for increasing numbers ofCAN nodes for coordinate spaces with different dimensions.For a system with nodes and dimensions,we see that the path length scales as in keeping with the analytical results for per-fectly partitioned coordinate spaces.Because increasing the number of dimensions implies that a node has more neighbors,the routing fault tolerance also improves as a node now has more potential next hop nodes along which messages can be routed in the event that one or more neighboring nodes crash.3.2Realities:multiple coordinate spacesThe second observation is that we can maintain multiple,inde-pendent coordinate spaces with each node in the system being as-signed a different zone in each coordinate space.We call each such coordinate space a“reality”.Hence,for a CAN with realities,a single node is assigned coordinate zones,one on every reality and holds independent neighbor sets.The contents of the hash table are replicated on every reality. This replication improves data availability.For example,say a pointer to a particularfile is to be stored at the coordinate loca-tion(x,y,z).With four independent realities,this pointer would be stored at four different nodes corresponding to the coordinates (x,y,z)on each reality and hence it is unavailable only when all four nodes are unavailable.Multiple realities also improve rout-ing fault tolerance,because in the case of a routing breakdown on one reality,messages can continue to be routed using the remaining realities.Further,because the contents of the hash table are replicated on every reality,routing to location(x,y,z)translates to reaching(x,y,z) on any reality.A given node owns one zone per reality each of which is at a distinct,and possibly distant,location in the coordi-nate space.Thus,an individual node has the ability to reach distant portions of the coordinate space in a single hop,thereby greatly re-ducing the average path length.To forward a message,a node now checks all its neighbors on each reality and forwards the message to that neighbor with coordinates closest to the destination.Figure5 plots the path length for increasing numbers of nodes for different numbers of realities.From the graph,we see that realities greatly reduce path length.Thus,using multiple realities reduces the path length and hence the overall CAN path latency.Multiple dimensions versus multiple realities Increasing either the number of dimensions or realities results in shorter path lengths,but higher per-node neighbor state and main-tenance traffic.Here we compare the relative improvements caused by each of these features.Figure6plots the path length versus the average number of neigh-bors maintained per node for increasing dimensions and realities. We see that for the same number of neighbors,increasing the di-mensions of the space yields shorter path lengths than increasing the number of realities.One should not,however,conclude from these tests that multiple dimensions are more valuable than multi-ple realities because multiple realities offer other benefits such as improved data availability and fault-tolerance.Rather,the point to take away is that if one were willing to incur an increase in the av-erage per-node neighbor state for the primary purpose of improving routing efficiency,then the right way to do so would be to increase the dimensionality of the coordinate space rather than the number of realities.3.3Better CAN routing metricsThe routing metric,as described in Section2.1,is the progress in terms of Cartesian distance made towards the destination.One can improve this metric to better reflect the underlying IP topology by having each node measure the network-level round-trip-time RTT to each of its neighbors.For a given destination,a message481632641282562561024409616K 64K 256K1MN u m b e r o f h o p sNumber of nodes#realities=12 dimensions3 dimensions4 dimensions5 dimensionsFigure 4:Effect of dimensions on path length 2481632641282562561024409616K 64K 256K1MN u m b e r o f h o p sNumber of nodes#dimensions=21 reality2 realities3 realities4 realitiesFigure 5:Effect of multiple realities on path length 10152025N u m b e r o f h o p sNumber of neighborsNumber of nodes = 131,072Figure 6:Path length with increasing neighbor stateis forwarded to the neighbor with the maximum ratio of progress to RTT.This favors lower latency paths,and helps the application level CAN routing avoid unnecessarily long hops.Unlike increasing the number of dimensions or realities,RTT-weighted routing aims at reducing the latency of individual hops along the path and not at reducing the path length.Thus,our metric for evaluating the efficacy of RTT-weighted routing is the per-hop latency,obtained by dividing the overall path latency by the path length.To quantify the effect of this routing metric,we used Transit-Stub topologies with link latencies of 100ms for intra-transit do-main links,10ms for stub-transit links and 1ms for intra-stub do-main links.With our simulated topology,the average end-to-end la-tency of the underlying IP network path between randomly selected source-destination nodes is approximately 115ms.Table 1com-pares the average per-hop latency with and without RTT weighting.These latencies were averaged over test runs with ,the number of nodes in the CAN,ranging from to .As can be seen,while the per-hop latency without RTT-weighted routing matches the underlying average IP network latency,RTT-weighted routing lowers the per-hop latency by between 24%and 40%depending on the number of dimensions.Higher dimensions give more next-hop forwarding choices and hence even greater im-provements.3.4Overloading coordinate zonesSo far,our design assumes that a zone is,at any point in time,assigned to a single node in the system.We now modify this to allow multiple nodes to share the same zone.Nodes that share the same zone are termed peers.We define a system parameter MAXPEERS ,which is the maximum number of allowable peers per zone (we imagine that this value would typically be rather low,3or 4for example).With zone overloading,a node maintains a list of its peers in ad-dition to its neighbor list.While a node must know all the peers in its own zone,it need not track all the peers in its neighboring zones.Rather,a node selects one neighbor from amongst the peers in each of its neighboring zones.Thus,zone overloading does not increase the amount of neighbor information an individual node must hold,but does require it to hold additional state for up to MAXPEERS peer nodes.Overloading a zone is achieved as follows:When a new node joins the system,it discovers,as before,an existent node whose zone it is meant to occupy.Rather than directly splitting its zoneas described earlier,node first checks whether it has fewer than MAXPEERS peer nodes.If so,the new node merely joins ’s zone without any space splitting.Node obtains both its peer list and its list of coordinate neighbors from .Periodic soft-state updates from serve to inform ’s peers and neighbors about its entry into the system.If the zone is full (already has MAXPEERS nodes),then the zone is split into half as before.Node informs each of the nodes on it’s peer-list that the space is to be ing a deterministic rule (for example the ordering of IP addresses),the nodes on the peer list together with the new node divide themselves equally between the two halves of the now split zone.As before,obtains its initial list of peers and neighbors from .Periodically,a node sends its coordinate neighbor a request for its list of peers,then measures the RTT to all the nodes in that neighboring zone and retains the node with the lowest RTT as its neighbor in that zone.Thus a node will,over time,measure the round-trip-time to all the nodes in each neighboring zone and retain the closest (i.e.lowest latency)nodes in its coordinate neighbor set.After its initial bootstrap into the system,a node can perform this RTT measurement operation at very infrequent intervals so as to not unnecessarily generate large amounts of control traffic.The contents of the hash table itself may be either divided or replicated across the nodes in a zone.Replication provides higher availability but increases the size of the data stored at every node by a factor of MAXPEERS (because the overall space is now partitioned into fewer,and hence larger,zones)and data consistency must be maintained across peer nodes.On the other hand,partitioning data among a set of peer nodes does not require consistency mechanisms or increased data storage but does not improve availability either.Overloading zones offers many advantages:reduced path length (number of hops),and hence reduced path latency,because placing multiple nodes per zone has the same effect as reducing the number of nodes in the system.reduced per-hop latency because a node now has multiple choices in its selection of neighboring nodes and can select neighbors that are closer in terms of latency.Table 2lists the average per-hop latency for increasing MAXPEERS for system sizes ranging from to nodes with the same Transit-Stub simulation topologies as in Section 3.3.We see that placing 4nodes per zone can reduce the per-hop latency by about 45%.improved fault tolerance because a zone is vacant only when。