Segmented information dispersal (SID) for efficient reconstruction in fault-tolerant video

Segmented Information Dispersal(SID)for Efficient Reconstruction in Fault-Tolerant Video ServersAriel Cohen Walter A.BurkhardGemini Storage Systems LaboratoryDepartment of Computer Science&EngineeringUniversity of California,San DiegoLa Jolla,CA92093-0114acohen@ burkhard@ABSTRACTThis paper presents a new data organization for disk arrays—Segmented Information Dispersal(SID).SID provides pro-tection against disk failures while ensuring that the recon-struction of the missing data requires only relatively small contiguous accesses to the available disks.SID has a num-ber of properties that make it an attractive solution for fault-tolerant video servers.Under fault-free conditions,SID per-forms as well as RAID5and organizations based on bal-anced incomplete block designs(BIBD).Under failure,SID performs much better than RAID5since it significantly re-duces the size of the disk accesses performed by the recon-struction process.SID also performs much better than BIBD by ensuring the contiguity of the reconstruction accesses. This contiguity property is a very significant factor for video retrieval workloads,as we demonstrate in this paper.We present SID data organizations with a concise representation which enables the reconstruction process to locate efficiently the needed video and check data.KEYWORDS:video server,fault-tolerance,segmented in-formation dispersalINTRODUCTIONAchieving fault-tolerance in disk arrays for video retrieval workloads is one of the challenging problems which need to be addressed by the designer of a large-scale video server. In this paper,we describe a novel data organization which provides an attractive solution for this problem.A storage server for digital video retrieval consists of an array of high-performance disks which is connected to display sta-tions(e.g.personal computers or set-top boxes)over a high-bandwidth network.The disk array may be attached to a tertiary storage device containing additional movies,a copy of the movies on the disk array,and possibly a copy of the check data(i.e.the data used for fault-tolerance purposes). The system uses memory either at the server or at the display stations(or both)for buffering.Naturally,the goal is to serve as many concurrent users as possible given the available re-sources.The workload experienced by the video server is essentially a read-only workload in which the task of the server is to retrieve movies from the disk array for consumption by the display units.Occasionally,a new movie may be copied from the tertiary storage device(or another server)to the disk array,and an existing movie may be erased.The main re-quirement for the server is to replenish the buffers in such a way that buffer starvation or overflow never occur for any of the streams which are being serviced.The buffers typically drain at a rate of1.5–40Mb/s(depending on the compression scheme and parameters used).Our goal is to devise a data organization for the disk array which achieves the following:a large number of concurrent streams,low buffering requirements,load balancing,fault-tolerance without adverse effects on fault-free perfor-mance,andreasonable performance under disk failure;this includes continued service when a disk fails(degraded mode)as well as reconstruction on a spare disk(rebuild mode).During degraded mode operation,data from surviving disks is used to reconstruct the data that is being requested from the failed disk;tertiary storage devices cannot assume responsi-bility for this real-time retrieval task.However,a disk rebuild may be performed by reading the data from a tertiary storage device,thus avoiding the need to increase the load on the surviving disks for the rebuild operation.BASIC PRINCIPLESData LayoutsThe disk array is devoted to video data and check data;sys-temfiles and metadata are stored separately.Movies are di-vided into equal size slices which serve as the basic unit ofaccess to the data;each slice is stored contiguously.The slice size is typically a few hundred KBs.A simple data organization without check data is shown inFigure 1.denotes slice of movie .This round-robin assignment of slices to disks achieves load balancing.If this scheme is used,a disk failure results in a complete loss of service.........................M1.1M1.5M1.6M1.7M1.8M1.2M1.3M1.4M2.1M2.5M2.2M2.6M2.3M2.7M2.4M2.8Figure 1:Slice striping for load balancingFigure 2shows a RAID 3organization [8].denotes fragment of slice of movie ,and denotes the par-ity of the fragments of slice of movie (e.g.).In this example,the disks are dividedinto two parity groups.The data within rectangles is read in parallel by all disks in the group.Note that each group can sustain up to one disk failure.Moreover,no degrada-tion of performance occurs if the parity can be computed fast enough.However,this organization suffers from poor fault-free performance due to the dedicated parity disks which are not utilized under fault-free operation,and the effects of the fine-grained striping (see [3]for a study of this).Figure 3shows a RAID 5organization [8].denotes slice of movie ,and denotes the parity of the slices that appear in the same row.Each of the two parity groups in this example can sustain up to one disk failure.This organiza-tion has good fault-free performance (see [3]),but it suffers from poor performance under failure since an affected par-ity group becomes a bottleneck:the load on the surviving disks doubles because of the need to perform the reconstruc-tion reads in addition to the normal workload.Our proposed organization (SID)addresses this issue.Read SchedulingIn this section we describe one possible read scheduling ap-proach which is based on reading cycles.Other approaches are possible with SID as well.Time during playback is divided into reading cycles during which exactly one slice is read for each stream from some disk (or group in the case of a RAID 3organization).A read-ing cycle concurrently serves a number of groups of streams which we call cohorts .Each stream is a member of exactly one cohort in each reading cycle.We can think of cohorts as tasks to be performed by a circular pipeline.During a read-ing cycle,each disk serves one cohort,and then the cohorts move to the next disk where they will be served during the next reading cycle.The maximum number of streams in a cohort is fixed to per-mit a certain fixed maximum number of streams to be ser-viced.When the number of streams in a cohort is lower thanthis maximum number,we say that the cohort contains one or more free slots .When the display of a new stream needs to be initiated,the system waits until a cohort with a free slot is about to be served by the group where the first slice of the requested movie resides;the new stream is then incorporated into this cohort.When a stream ends,it is dropped from its cohort;this results in a free slot which can be used to initi-ate a new stream.Note that once a stream is assigned to a cohort,it remains a member of that cohort until its display is finished.Figure 4shows examples of reading cycles along with their cohorts in a server with two disks.In this example,a cohort may contain up to 4streams.Cohort 0is served by disk 0during reading cycle ,disk 1during the reading cycle,and disk 0again during reading cycle.Cohort 1is served by disk 1during reading cycle ,disk 0during readingcycle,and disk 1during reading cycle .A new stream ()is incorporated into cohort 1during reading cycle.Note that the order of reads may vary from reading cycle to reading cycle;this flexibility enables us to use seek optimization algorithms.SSS567SSSSSSS7SSSSSSS142376556SS884321SSSS4213Disk 0Disk 1reading cycle reading cycle reading cycle Cohort 0Cohort 1Cohort 0Cohort 1Cohort 0Cohort 1tt+1t+2Figure 4:Reading cyclesDuring each reading cycle,for each stream,the system reads the next slice from the group serving the stream’s cohort into the buffer while consuming the slice which was read in the previous reading cycle.Let the slice display time be (sec-onds);if the reading cycle takes less than seconds,the sys-tem waits until seconds are over before starting the next reading cycle.The maximum amount of time between the retrievals of suc-cessive slices for a stream is called the replenishment latency .Figure 5illustrates this latency.The replenishment latency determines the amount of data that needs to be buffered per stream;hence,reducing this latency is the way to reduce the buffering requirements or increase the number of supportable streams for a give amount of buffering.The replenishment latency is determined by the number of streams in a cohort,seek times,rotational latencies,and slice transfer times.RELATED WORKParity-based schemes for fault-tolerant video servers were studied by Berson,Golubchik and Muntz [1].They stud-ied an approach which is similar under fault-free conditions to the RAID 5approach described above.Under failure,a....................................M1.2.1M1.2.2P1.2M2.4.1M2.4.2P2.4M1.4.1M1.4.2P1.4M2.2.1M2.2.2P2.2M1.2M1.4M2.2M2.4M1.1.1M1.1.2P1.1M2.3.1M2.3.2P2.3M1.3.1M1.3.2P1.3M2.1.1M2.1.2P2.1M1.1M1.3M2.1M2.3Figure 2:RAID 3..................P P P P P PM1.3M1.4M1.7M1.8M1.12M1.11M2.3M2.4M2.7M2.8M2.12M2.11P P P P P PM1.1M1.2M1.5M1.6M1.9M1.10M2.1M2.2M2.5M2.6M2.9M2.10Figure 3:RAID 5reading cycle reading cycle replenishment latency12343241S S S S S S S S Figure 5:Replenishment latencydifferent read scheduling policy is employed for affected par-ity groups:data from all surviving disks in the parity group(i.e.an entire parity stripe which contains a number of video slices and parity data)is read for a stream during a single reading cycle,and this data is consumed during a number of reading cycles.This obviates the need to perform sep-arate reconstruction reads since the data needed to perform the reconstruction is already stored in the buffer.Hence,the load increase incurred by the surviving disks is minimized by exploiting the sequentiality of video playback.However,significant additional buffering for degraded mode operation is required.A disadvantage of this scheme is that there is a transition phase as the parity group switches to degraded mode;during this phase,some of the data is not delivered and consequently “hiccups”occur.The scheme proposed in this paper (SID)provides a smooth transition to degraded mode.Vin,Shenoy and Rao [11]proposed a method which does not rely on error-correcting codes such as parity,but on prop-erties of the video data itself.Their technique is a combi-nation of a compression algorithm and a disk array layout.The compression algorithm partitions the data in such a way that an acceptable (but not perfect)reconstruction of an im-age can be performed when some of the data is not avail-able.The disk array layout ensures that the recovery pro-cess does not impose any additional load on the surviving disks.This method is based on a non-standard compressionscheme which is an adaptation of the JPEG image compres-sion technique which does not exploit the temporal redun-dancy of motion video,and thus it results in poor compres-sion ratios when compared to standard motion video com-pression schemes such as MPEG (note that motion prediction accounts for most of MPEG’s ability to achieve good quality at a low bit-rate.)Such lower compression ratios result in a need to maintain a higher bit-rate which has a detrimental effect on the performance of the server and the number of movies that can be stored.The technique proposed in this paper is oblivious to the nature of the data stored in the disk array,so it works well with MPEG and other standard motion video compression schemes.Parity declustering [5]is a redundancy scheme for reducing the additional load on the surviving disks during the recon-struction of the content of a failed disk.This scheme is tar-geted at transaction processing environments;it is not well-suited for video retrieval workloads.This issue is discussed further in another section below (SID vs.Balanced Incom-plete Block Designs).GOALSOur goals in developing a new data organization for fault-tolerant disk arrays for video are as follows:we desire to obtain RAID 5-like performance under fault-free conditions,reduce the reconstruction load under disk failure while main-taining the access contiguity of the read operations,and pro-vide optimized reconstruction of video data rather than obliv-ious video and check data reconstruction.The last goal is motivated by the fact that only the video data is needed for continued service under failure,and the effi-cient delivery of this data is critical to maintaining reason-able performance under failure.The check data,on the otherhand,is less time-critical—it is needed only for the rebuild process.Since the video retrieval workload is essentially a read-only workload,the rebuild process may even be per-formed by copying the data which existed on the failed disk from tertiary storage without engaging any of the surviving disks.SID achieves the goal of load reduction by reducing the size of the reconstruction reads;this reduces the transfer time which is the dominant component of the replenishment la-tency.The load reduction is illustrated in Figure 6.In thisexample,there are three streams per cohort.,,and denote the three slices read for a cohort during a reading cy-cle.denotes a read performed for reconstruction purposes.When a RAID 5organization is used,the size of the recon-struction reads is the same as the size of the slice reads (re-constructing a slice of bytes requires reading bytes from each of the surviving disks in the parity group.)Hence,a reading cycle under failure is approximately twice as long as a reading cycle under fault-free conditions.With SID,the size of the reconstruction reads will be smaller than the size of the slice reads;this will result in shorter reading cycles.SID under failure:3S RAID 5 under failure:RAID 5/SID fault-free:S 3S 1reading cycle reading cyclereplenishment latencyS 2S 2S 3S 11S S reading cycle S 322S 1S reading cycle replenishment latencyS S 3S 12S S S 231replenishment latencyreading cyclereading cycle RRRRRRR R R R R R Figure 6:Load reduction through smaller reconstruc-tion readsWe can state the problem that we wish to address as fol-lows (see Figure 7):we are given disks containing video data,and we wish to add some check data to each disk (the check data will occupy contiguous spare tracks at the begin-ning or end of the disks)such that the following holds:the data (video and check)is protected against a single disk fail-ure,and the recovery of a slice of bytes of video data (is known in advance)from a failed disk requires reading atmostcontiguous bytes from any surviving disk (we call this property the limited contiguous access length (LCAL)property.)is called the dispersal factor of the data orga-nization.The ratio of the amount of check data to the total amount of data (check and video)is called the redundancy rate of the solution.. . .DataD D D 12n. . .12nC 1C 2C nDataCheckD D D Figure 7:We wish to add check data to the data con-tent of the disks such that protection against a single disk failure is obtained and the LCAL property holds.The goal of SID is to allow reconstruction while requiring only small accesses to the disks.Thus,we wish to maximize the dispersal factor ;on the other hand,we wish to min-imize the redundancy rate.The latter goal runs counter to the former goal,of course.Hence,the problem that needs to be addressed is that of finding a solution which strikes a good balance between the obtained dispersal factor and the required level of redundancy.An obvious upper bound on is.In fact,it is very easy to construct solutions that achieve this upper bound.Fig-ure 8shows an example of such a solution.In this example,each of the disks is divided into four equal size logical seg-ments (the figure only illustrates the segments for disk ;the data belonging to the four segments of disk is de-noted by four different shades of gray.)The segments are -interleaved where is the slice size (-interleaving is used in the general case.)The check portion of each disk contains one segment from each of the other disks (the figureonly shows the locations of the data of disk;the data of the other disks is distributed similarly.)If a slice of bytes on a failed disk needs to be reconstructed,its content can be found in the check portions of the other disks.Each of these disks will provide one fourth of the data,and the recon-struction reads will be contiguous;hence,the LCAL property holds.The disadvantage of this construction is the amount of redundancy that it requires:one half of the storage space is devoted to check data.RAID 5organizations are on the other extreme of the range of solutions.The obtained dispersal factor is 1,while the redundancy rate is ,which is the minimum required for single erasure correction.SEGMENTED INFORMATION DISPERSAL (SID)SID provides a middle ground between the two extremes of duplication and RAID 5.Figure 9shows an example of a SID solution for 5disks which achieves a dispersal factorof 2with a redundancy rate of.In this example,each of the disks is divided into two equal size logical segments(e.g.is divided into and ).The segments are -interleaved where is the slice size (e.g.the first bytes of are in ,the second set of bytes is inDataCheckD D D D D 12354(r bytes)video sliceFigure 8:Highest dispersal by duplication:,,redundancy=1/2,the third set is in ,the fourth is in ,etc.)Note that the sole purpose of the segments is data labeling;no datais rearranged.The-interleaving ensures that a contiguous video slice of bytes will contain equal amounts of data from the two segments of the disk (recall that the disk array is dedicated to video data and check data).2-interleaved)r(=====C 1S 5,1S 2,2S 3,2S 1,1C 2C 32,1S S 4,2S 3,1C 4C 5S 4,11,2S S 5,2D D D D D 12345DataCheckFigure 9:SID layout:,,redundancy=1/3In Figure 9,the content of the check portions of the disks was chosen in such a way that when a video slice from any failed disk needs to be reconstructed,the two halves of the data will be reconstructed from two disjoint pairs of disks.Hence,a dispersal factor of 2is obtained.The size of thecheck portion of each disk in this example is the same as the size of a segment (which is half the size of the data on thedisk);therefore the obtained redundancy rate is.The general problem is illustrated in Figure 10.We are given the video data content of disks,a video slice size ,and a dispersal factor (the set of feasible values for depends on ,of course;we return to this issue below.)Each disk con-tains a number of contiguous video slices.The data on eachdisk is labeled by dividing each disk into-interleaved logical segments (we assume that divides .)For example,in Figure 9each disk contains three video slices of size bytes,and each disk is divided into two -interleaved log-ical segments of size bytes.The goal is to add a check portion to each disk such that contains the XOR of data segments that belong to other disks.The pur-pose of the check data is to enable the recovery of the datain a single failed disk;that is,all the video data in should be recoverable as long as all other disks are available.In addition,the recovery process must have the following special property:during the recovery of any videoslice which belongs to a failed disk,no more thanbytes must be read from any of the surviving disks,and these bytes must be contiguous.This property of the recovery process is the limited contiguous access length (LCAL)property.Theratiois called the redundancy rate of the layout.Note that the LCAL property applies only to the reconstruc-tion of the data portion of a missing fragment.It is possibleto impose a stronger condition and require the recovery pro-cess to satisfy the LCAL property when both the data and the check area of a missing fragment are reconstructed.This problem is discussed in [2].The problem of SID layouts for multiple failures is also discussed in [2].Figure 11shows a larger example with 10disks and a disper-sal factor of 3.When a video slice of bytes from any disk needs to be reconstructed,its three equal size parts (the ones belonging to the three segments of the failed disk)will be reconstructed from three disjoint triples of surviving disks.Hence,each surviving disk will be required to read only bytes.Given a certain number of disks ,we naturally wish to ob-tain a SID data organization with the highest possible disper-sal factor .This maximizes the load reduction and mini-mizes the redundancy rate (recall that the redundancy rate is).It is easy to see that .This is a result ofthe LCAL property:when a slice of size bytes needs to be reconstructed,bytes are read from the check portion of disks,and bytes are read from the data portion ofother disks;all these disks must be different to maintain the LCAL property,so we get a total ofdisks.Therefore,an upper bound on is.In [2]we show that this upper bound can be obtained only for...............qrqr 1D 2D n D . . .n S S S S S S S S S 1,12,1n,11,22,2n,21,q 2,q n,q12q-interleaved . . .12qData Check-interleaved )(D 1D 2D Figure 10:SID layout:=XOR ofdata segments that belong to otherdisksD D D Figure 11:SID layout:,,redundancy=very few values of (,and possibly 57).However,using construction methods which are described in [2],it is possible to construct SID solutions withTable1:SID solutions forsolution solution solution 25725725725(6)72582(3)573673673673683683683683(4)6(7)8468468468468468468478468469478478578578578579578(9)578578the parity group size is 3.By interleaving the parity groups on the disks in a manner similar to that of SID’s segments,we obtain a dispersal factor of.For example,if a video slice of bytes from disk C needs to be reconstructed,onlybytes will be read from each of the otherdisks.Data + Check123124234314D 4D D D 123Figure 12:BIBD layout:dispersal factor=,redundancy=Note that although a dispersal factor of is obtained by the layout of Figure 12,the LCAL property does not hold since the reconstruction reads are not contiguous!Contigu-ous reconstruction accesses cannot be obtained by layouts based on this block design no matter how the parity groups are permuted within the disks.There are few block designs that result in a layout which enables contiguous reconstruc-tion (these are block designs with and a few block designs with ;see [4]for a definition of balanced in-complete block designs and .)There are even fewer block designs that enable contiguous reconstruction while provid-ing acceptable redundancy rates and dispersal factors.SID,on the other hand,provides good solutions for any reasonable number of disks (see Table 1).In the next section,we will see that the inability to provide contiguous reconstruction accesses is a serious deficiency of BIBD layouts for video workloads.This problem is of lesser concern in transaction processing environments since the crit-ical issue for those workloads is the aggregate load increase over many transactions.In the example of Figure 12,one quarter of the transactions would be reconstructed from each of the parity groups;hence,each surviving disk would par-ticipate in the reconstruction of only two thirds of the trans-actions.For video workloads,on the other hand,the most important measure is the length of a reading cycle which is determined by the time it takes to retrieve a few video slices;hence,we need to minimize the reconstruction time per slice.PERFORMANCE RESULTSWe conducted an analytical study which compares the per-formance of various possible data organizations.The diskparameters were based on the performance characteristics of Seagate Elite 9disks.A description of the model and the parameters used can be found in [2].Figure 13shows how the buffering requirement per stream varies with the total number of concurrent streams for a disk array with 12disks and a video consumption rate of 4Mb/s.The figure compares the performance of three data organiza-outs has SID ure,per ure Total Number of StreamsB u f f e r S i z e p e r S t r e a m (K B )Figure 13:SID vs.RAID 3&5:,,redundancy=,video consumption=4Mb/sNote,however,that both RAID 3and RAID 5provide a higher degree of fault-tolerance in this case since they protect against one disk failure per parity group while SID protects against one failure in the entire disk array.For a fixed level of fault-tolerance,SID requires a higher redundancy rate than RAID 3or RAID 5,but it provides a significantly higher level of performance under failure.The same level of fault-tolerance as that of RAID 3or RAID 5can be achieved by dividing the disk array into SID groups in the same man-ner that RAID 3and RAID 5arrays are divided into parity groups.Since a large number of disks is desirable for perfor-mance reasons regardless of the data organization used,and since current and future disks offer a very high capacity,trad-ing some amount of storage space for a significantly higher level of performance under disk failure is very acceptable.Total Number of StreamsBufferSizeperStream(KB)RAID 5 with failureRAID 3 with/without failureSID with failureRAID 5/SID without failureFigure14:SID vs.RAID3&5:,,redundancy=,video consumption=4Mb/sFigure14shows the performance of a disk array with90disks.The redundancy rate for all data organizations is(i.e.the RAID3and RAID5layouts consist of10paritygroups of size9,and the SID layout has a dispersal factorof8.)The impact of using SID in this case is considerablylarger than in the previous case because of the higher disper-sal factor.For example,if we limit the buffer size per streamto1MB,SID can support540streams under failure,whileRAID5can only support360streams.The impact of the dispersal factor is illustrated in Figure15.We see,as expected,that performance improves as the dis-persal factor increases,but diminishing returns are obtainedonce the dispersal factor is high enough to make latenciesother than the transfer time dominate.As mentioned in the SID vs.Balanced Incomplete BlockDesigns section,the most important advantage of SID whencompared to BIBD layouts is the guarantee of contiguousreconstruction reads.Figure16shows the impact of discon-tiguity on the performance.The different curves correspondto different numbers of accesses performed during the recon-struction of a video slice.With SID,only one access is per-formed,but we studied the performance for larger numbersof accesses in order to determine the impact of the disconti-guity exhibited by BIBD layouts.We see that even a smallamount of discontiguity seriously affects the performance.Once the number of accesses increases beyond4,the per-formance becomes even worse than that of RAID5(whichrequires reading larger,but contiguous,pieces ofdata).Number of Streams per DiskRAID 5 with failureSID (q=2)SID (q=4)SID (q=8)SID (q=16)SID (q=256)RAID 5/SID without failureFigure15:Dispersal factor impact:video consump-tion=4Mb/sCONCLUSIONSSID achieves a significant reduction in the size of the recon-struction accesses while maintaining the contiguity of theseaccesses.It strikes a good balance between the obtained dis-persal factor and the required level of redundancy.RAID5organizations require reconstruction accesses which are aslarge as the size of the data to be reconstructed.BIBD or-ganizations achieve a size reduction,but do not generallymaintain contiguity.SID is an attractive data organizationfor fault-tolerant disk arrays when the reconstruction pro-ceeds in units of a predeterminedfixed size.Video workloadshave this property(the reconstruction unit is a video slice.)Our performance study shows the significantly higher levelof performance under failure obtained by SID when com-pared to RAID5or BIBD.Under fault-free conditions,SIDperforms as well as RAID5or BIBD.ACKNOWLEDGEMENTSThis work was supported in part by grants from SymbiosLogic,Wichita,Kansas,and the University of California MI-CRO program.REFERENCES1.Berson,S.,L.Golubchik,and R.R.Muntz.Fault Tol-erant Design of Multimedia Servers.SIGMOD Record,24(2),pp.364–375,1995.2.Cohen, A.Segmented Information Dispersal.Ph.D.Dissertation,Department of Computer Science&En-。