重复数据删除PPT-谭玉娟

Data Deduplication (重复数据删除)谭玉娟 华中科技大学Data DeduplicationMotivation Background Research Topic Use Cases How to Use ？Motivation (1)Global StorageIDC: 3/4Motivation (2)Backup SystemMotivation(3)Data transfer bottleneckBackup p 1 TB data to Amazon S3The average bandwidth measured is 800KB/s1 × 1012Bytes⎛ 800 ⎜ ⎝× 1013Bytessecond⎞ ⎟ ⎠= 1,250,000secondsmore than 14 days unacceptable backup window2003-2008: Wide area: 2.7xComputing 16x disk storage: 10xMotivation (4)Motivation (5)Dedup Background Whole file Deduplicationfoo01101010….. ….110010101bar01101010 01101010…..….110010101 110010101Dedup Background Fixed Chunk Deduplicationfoo01101010….. ….110010101 …11101111111bar01101010…..….110010101 … 110010101Dedup BackgroundFixed Chunk…Insert Data…V i bl Si Variable Sized d Ch Chunk k…Insert Data…Dedup Background Rabin Figerprintingfoo00100000….. 1 01101011….. 00100000…..110101 101010 010100bar01101011….. 01101010…..The Deduplication SpaceAlgorithm Parameters CostDeduplication effectivenessWhole-fileLowLowestSeeksFixed Chunk Chunk SizeCPU Complexity SeeksMiddleRabin fingerprintsAverage Chunk SizeMore CPU More Complexity p yHighestDeduplication VS Compression DeduplicationLossless compression Granularity: File-level, Chunk-level Large scale storage system y compression p technology gy SystemCon entional lossless compression ConventionalGranularity: Byte-level Small datasets General data compression technologyImplementation(1)Client-Side: used for saving bandwdithDeduplicationBackup stream after deduplicationApplication ServerStorage g deviceImplementation(2)Target-Side: used for saving StorageDeduplicationBackup stream before deduplicationApplication ServerStorage deviceDeduplication ProcessDeduplication ProcessChunking Indexing Index lookupLink data generationCommit new data chunkUpdate index tableResearch—磁盘瓶颈没有足够RAM 空间存放所有数据块的索引信息， 大部分的索引信息必须存放在磁盘上。

8TB 数据 20GB 索引. 800TB 数据 ,2TB 2TB 索引.T l Too large!! !!在重复数据块查找的过程中 索引的查询会带来大 在重复数据块查找的过程中，索引的查询会带来大 量的磁盘访问，引来了重复数据删除过程中的磁盘 瓶颈问题。

瓶颈问题Research—可靠性文件 1 文件 2 文件 3数据块 1数据块 2数据块 3数据块 4数据块 5数据块 6可靠性Research—读性能文件 1 文件 2 文件 3数据块 1数据块 2数据块 3数据块 4数据块 5数据块 6读性能研究热点—能耗问题文件 1 文件 2 文件 3数据块 1数据块 2数据块 3数据块 4数据块 5数据块 6能耗问题实例系统—VentiVenti是一个数据归档存储系统。

它首次将重 复数据删除应用于备份归档系统中。

复数据删除应用于备份归档系统中 特点： 特点使用固定长度分块 使用write-once写策略 使用缓冲提高吞吐率 使用缓冲提高吞 率实例系统—Venti索引表备份归档系统—VentiVenti原型系统方框图通过使用缓冲，系统吞吐率从5.6MB/s 提升到6.4MB /s左右备份归档系统—DDFSDDFS是Data Domain公司开发的重复数据删除文 件系统。

用于基于磁盘的备份存储系统中。

主要关注如何提高重复数据删除的吞吐率，使备份 速度能达到100MB/s以上. 采用如下三种技术：Summary Vector Stream-Informed segment layout (SISL) Locality Preserved Caching (LPC)实例系统—DDFS实例系统—DDFS实例系统—DDFS实例系统—DDFS实例系统—DDFS磁盘访问次数的减少通过使用Summary Vector 和 LPC，磁盘I/O次数减少了99%实例系统—DDFS吞吐率单数据流达到113MB/s，四数据流达到 四数据流达到217MB/SWhy study deduplication?$0.046 $0 046 per GB pImproved sequential bandwidth Reduced per-byte cost9ms per seekPerformance Impacts Data WritesThe comparison of fingerprints Disk bottleneckData ReadsDe-linearizing data placement Random disk seeksData ReliabilityWhen do we exploit duplicates? It DependsHow much can you get back from deduping? How does fragmenting files affect performance? f ? How often will you access the data?Trade off (Trading away the sequentiality for spacesavings)Embarrassingly compressibleBackups virtual desktops Backups,General purpose datasets?Pi Primary St Storage system? t ?A Study of Practical Deduplication FAST11’s Best Paper Dt Dtasets t857 desktop computers (file systems) 162 terabytes Over 4 weeksAnalysisRedundancy in File Contents MetadataWhat is the relative deduplication rate of the algorithms? Dedup p by y method and chunk size18%-20%10%-11%How does the number of file systems influence deduplication? Dedup by file system count42% 39% 20%What if I was doing full weekly backups? Backup dedup over 4 weeks82%74%67%Deduplication RateLive File SystemThe Whole file deduplication and sparseness can remove 75% of the space savings by Rabin fingerprints. fingerprintsBackup imagesThe Whole file deduplication can remove 87% of the space savings by Rabin fingerprints.Which of these types deduplicate e well? Whole-file duplicatesWhat files make up p the 20% difference between whole file dedup and sparse file as compared to more aggressive file, deduplication?Where does fine granularity help?What types of files take up disk space? Disk consumption by file typeConclusion The whole-file deduplication together with sparseness is a highly efficient means of lowering storage consumption, even in a b k scenario. backup i The whole-file deduplication approaches the effectiveness of conventional deduplication at a much lower cost in performance and complexity.Reading ListAvoiding the Disk Bottleneck in the Data Domain Deduplication File System. In FAST 2008 conference. Cumulus Filesystem y Backup p to the Cloud. In FAST 2009 conference. HYDRAstor A Scalable Secondary Storage. In FAST 2009 conference. Sparse Indexing Large Scale, Inline Deduplication Using Sampling and Locality. In FAST 2009 conference Bimodal Content Defined Chunking for Backup Streams. In FAST 2010 conference. IO Deduplication Utilizing Content Similarity to Improve IO Performance. In FAST 2010 conference A Study of Practical Deduplication. In FAST 2011 conference Tradeoffs in Scalable Data Routing for Deduplication Clusters. In FAST 2011 conference Decentralized Deduplication in SAN Cluster File Systems. In USENIX ATC 2009 conference ChunkStash Speeding up Inline Storage Deduplication using Flash Memory. In USENIX ATC 2010 conference f Extreme Binning Scalable, Parallel Deduplication for Chunk-based File. In MASCOTS 2009 conference. Providing High Reliability in a Minimum Redundancy Archival Storage System System. In MASCOTS 2006 conference.Any Question ?Thanks ☺。