UDF文件系统介绍——Introduction to Universal Disk Format

Wenguang's Introduction to Universal Disk Format (UDF)NoteBecause of changes of MobileMe policy, this page can no longer be edited after July 2009. Links to new versions of this page will be posted in the "External Links" section of the Wikipedia article about UDF.Outline1. What is UDF?2. Why UDF?3. History of UDF Revisions4. Structure of the UDF Standard5. UDF Specification Tutorial5.1 Highlight of the UDF Format5.2 UDF Volume Structure and Mount Procedure5.3 UDF Partition Structure5.4 UDF File and Directory Structure5.5 Some UDF Terminologies5.6 Tips and Notes That Are Not in the UDF Standards1. What is UDF?Universal Disk Format (UDF) is a file system specification defined by OSTA. One objective of UDF is to replace the ISO9660 file system on optical media (CDs, DVDs, etc). It is also a good file system to replace FAT on removable media.2. Why UDF?Any removable media (CD, DVD, flash drive, external hard drive, etc) needs a file system format. Ideally, this format should have these characteristics: Can be understood by different platforms. This makes it possible to copy files between Windows, Mac, and Unix systems. FAT and ISO9660 are twoformats that can be understood by most systems. However, they have manylimitations.Its specification is open. ISO9660 is an open standard, while FAT belongs to Microsoft.Has rich features (preferably a super set of all common file systems) so information won't be lost when files are copied to this file system.Can support different kinds of physical media. Optical media is verydifferent from hard drives. Some media is write once (CD-R, DVD-R, DVD+R, BD-R), some needs defect-management (CD-RW, DVD-RW, DVD+RW, BD-RE, etc), some needs to be expanded sequentially before being overwritten (most RW media).Its format should be as simple as possible. This is important when thisformat is implemented in embedded devices (DVD player, Camcorder, Camera, etc). Complex data structures such as B-tree are not good candidates for this purpose.Its format should evolve in a compatible way so old media can still beaccessed by new systems.UDF is the only file system that meets all these standards, since it was designed for the information exchange purpose.UDF is an open standard.The design and evolution of UDF keeps compatibility in mind.UDF natively supports many modern file systems features:Large partition size (maximum 2TB with 512B block size, or 8TB with2KB block size)64-bit file sizeExtended attributes (e.g., named streams, or forks) without sizelimitationLong file names (maximum 254 bytes, any character can appear in thename)Unicode encoding of file namesSparse fileHard linksSymbolic linksMetadata checksumMetadata redundancy (optional in UDF 2.50 or later in metadatapartition)Defect management (for media that does not manage defect internally,such as CD-RW, DVD-RW, and DVD+RW)UDF defines how different platforms interact with each other. For example, it defines how to store Mac Finder Info and Resource Fork, NTFS ACL, UNIX ACL, OS/2 EA, etc. It also requires platforms to preserve the information that they don't understand.UDF is a truly universal file system. It can be used on all kinds ofoptical media, including read only (CD-ROM, DVD-ROM, BD-ROM (Blu-ray Disc Read-Only)), write once (CD-R, DVD-R, DVD+R, BD-R), rewritable (CD-RW,DVD-RW, DVD+RW, DVD-RAM, CD-MRW, DVD+MRW, BD-RE), and of course blockdevice (hard drives). Even write-once media appears as a big overwritable floppy under UDF.Drawbacks of the current revision of UDF as of 2.60:Limited partition size. 32-bit block number limits the partition size to 2TB for 512 sector size. Although it is not a problem for the currentoptical media, it may become a problem later.Does not provide a fast crash recovery mechanism. As the size of the media increases, crash recovery becomes more and more important. Full disc scan before mount becomes less feasible on slow optical media with tens ofgigabytes space. Although an implementation may use a journal to protect metadata integrity, this does not guarantee interoperability betweenplatforms since it is not part of the standard.Does not support compressed/encrypted file and directories. As device gets bigger and bigger, compression is not that important. However, encryption may become more compelling since UDF is mainly used on removable media.Becomes more and more complex. UDF 2.50 adds the metadata partition inorder to improve performance. File system metadata are clustered withinthe metadata partition so that they can be accessed quickly. Optionally, a mirror of the metadata partition could be duplicated to provide betterrobustness in a big cost of performance. This adds non-trivial complexity to the file system. Does the benefit of metadata partition warrant itscomplexity? If the UDF implementation organizes metadata properly, it may achieve similar (or better) performance to what the metadata partition can provide. Unfortunately, in UDF 2.50 or later, the use of metadatapartition is mandatory on overwrite media like CD-RW and hard drive. UDF2.60 even requires the use of metadata partition on write-once media usingpseudo-overwrite partition. If a UDF implementation wants to avoid thecomplexity of metadata partition, it should use UDF 2.00/2.01.Is not as popular as FAT and ISO9660 now. As more and more systemsimplement UDF, this problem will go away.3. History of UDF RevisionUDF is an evolving standard. Their major features are summarized in the following table.Summary of UDF Revision HistoryRevisionDatePublishedMajor New Features1.0208/1996First revision, suitable for read-only media1.5002/1997Support write once media and defect management on media2.0004/1998Support named streams2.0103/2000Fix minor errors2.5004/2003Support metadata partition for better performance2.6003/2005Support pseudo-overwrite partitionThere have been 6 UDF revisions published: 1.02, 1.50, 2.00, 2.01, 2.50, and 2.60. Revision 2.00 and 2.01 is very similar, and revision 2.50 and 2.60 is very similar. So there are four generations of UDF: 1.02, 1.50, 2.00/2.01,2.50/2.60. They are discussed in more details below.UDF 1.02 is the first UDF revision. It is the standard used by DVD movie.It is suitable for read-only and hard-drive like media.UDF 1.50 adds virtual partition and sparable partition. Virtual partition allows a write-once media (CD-R, DVD-R and DVD+R) appears as anoverwritable media. A write-once media appears as an overwritable floppy (but hundreds or thousands times larger), except that its available space keeps decreasing as you use it. Even removing files cannot reclaim space.The sparable partition performs defect management on the media, similar to what the hard drive firmware does on modern hard drives. This is because overwritable media such as CD-RW, DVD-RW, and DVD+RW can only beoverwritten for a limited number of times (several thousand times) andwill fail. A sparable partition makes a disc with many defects appear as a good one with a contiguous logical space.UDF 2.00 adds named streams to files and directories, as well as systemstreams to the logical volume. Named streams can be used to implementextended attributes in other file systems, such as the resource fork andACL in Mac OS X, and the ACL in NTFS. At the same time, the format of the mapping table for virtual partition is changed.UDF 2.01 fixed a few minor errors of 2.00 and does not introduce majorfeatures.UDF 2.50 brings in metadata partition, and increases the complexity of UDF to a new level. The metadata partition contains all metadata such asdirectories and blocks managing file space allocations. The objective of metadata partition is to improve file system performance by aggregatingmetadata together. The metadata partition could optionally supportsoftware mirroring so two copies of the metadata are maintained. Thisfeature pay a price on performance and improves the robustness of the file system, while at the same time makes the file system even more complex.This revision is the standard for the coming high-definition DVDs (HD-DVD or Blu-ray).UDF 2.60 adds the support for pseudo-overwritable partition when the drive supports pseudo-overwrite mode for write-once media. Pseudo-overwritemeans the drive manages a logical to physical address mapping (similar to virtual partition) so the file system can simply treat the partition asoverwritable. With the intention of reducing the file system complexity, UDF defines that some drives may not support pseudo-overwritablepartition, so the file system must use virtual partition to manage suchmedia. So in a long time when two types of drives co-exist, the filesystem must be able to handle both drives and thus will be even morecomplex, ironically.4. Structure of the UDF StandardThe UDF Standard contains two sets of specifications, ECMA-167 and UDF.ECMA (European Computer Manufacturer's Association) is a standards bodythat determines standards for computing technology. ECMA-167 is a volume and file structure standard. It is also ISO/IEC 13346. ECMA-167 defines a general volume and file format mainly targeted optical media (write-once and rewritable media). ECMA is defined as a general framework. It leaves enough choices and undecided details that need to be filled by anotherstandard. ECMA-167 has second edition (ECMA-167/2) and third edition(ECMA-167/3). ECMA-167/3 added named streams. So UDF revision 1.x is based on ECMA-167/2 while UDF revision 2.x is based on ECMA-167/3. ECMA-167/3 is almost identical to ECMA-167/2 except the named stream support, a higher revision number, and a few other minor places. These differences arediscussed in detail in ECMA-167/3 if you search the term "167/2".UDF is the standard defined by OSTA (Optical Storage TechnologyAssociation). UDF is based on the ECMA-167 framework, filling in all the necessary details, clarify ambiguous pieces.Because UDF consists of ECMA-167 and UDF, you need to have both standards in hand and read them side-by-side. To make things very clear, the style of the standard is like a reference book. Learning knowledge from a reference book is not fun. It is like learning a language by reading its dictionary from A to Z, and put all the grammar together by connecting all the fragments in the dictionary. Reading two standards is twice as worse: you need to learn two new languages A and B from two dictionaries, while dictionary B is written using language A. Another side effect of reading a standard is that it makes peoplefall asleep fairly quickly :-;The learning process should be iterative. You start reading ECMA-167 to get some feeling, and read UDF for corresponding sessions to get more feelings, and go back. At some point, grab a UDF disc and dump its structure and read what's on it, to verify what is in your mind. You don't need to finish readingECMA-167 and fully understands it before read UDF, because there are many details in ECMA-167 that are not used in UDF.The UDF tutorial in the following session explains what UDF looks like. I don't assume you have knowledge of other file systems such as the Unix file system. But if you know that, it will be easier to understand UDF.5. UDF Specification Tutorial5.1 Highlight of the UDF FormatCompared with the Unix file system (UFS/FFS/ext2), UDF's main structures are highlighted below.Where UDF is similar to FFSAn inode is used to represent a file or a directory. The UDF's termof inode is File Entry. In UDF 2.x, there is also Extended FileEntry which works in the same way as a File Entry, except that itsupports named streams. When we mention File Entry below, it meansthe File Entry in general, including File Entry and Extended FileEntry.A directory is a special file which contains many variable sizedirectory entries. Each entry has a variable size file name and theaddress of the File Entry (i.e., inode) of the file or directory. Thestructure of the directory is linear. A linear search is required tolookup a file with its name.Hard links are implemented by letting more than one directory entriespoint to the same File Entry (i.e., inode). Each File Entry maintainsa link count, but does not have information about which directoryentry points to it.Symbolic link is a special file containing a path.A bitmap is used to manage the free space of the file system,although read-only and write-once media do not use bitmaps.Where UDF is different from FFSEach File Entry (i.e., inode) consumes one disk block (512B on mosthard drives, and 2KB on most optical media). The File Entry isidentified by its block address. Unlike FFS, there is no limitationon number of File Entries in the file system.Small files (and directories) can be stored in the File Entry blockitself, similar to the embedded files in NTFS.An Extended File Entry inode block can point to another File Entrycalled a named steam directory, which may contain unlimited number ofnamed streams.Disk space allocated to a file/directory is managed by extents.Sparse file is supported by marking an extent as sparse. For fileswith many fragments that all its extents cannot be stored in the FileEntry block, more disk blocks can be allocated to store the extentinformation. These disk blocks are linked together as a singly linkedlist.The file structure of UDF is not built on the raw device, but on thepartitions. UDF has the most complex partition management amongexisting file systems. Although different partition (type 1,sparable, virtual, metadata, and pseudo-overwritable) has differentunderlying physical properties, they provide an almost universalinterface to the above layer (the file and directory structures):each partition is a contiguous logical space consisting of blockswith logical numbers from 0 to n-1, where n is the partition size. 5.2 UDF Volume Structure and Mount ProcedureThe volume structure is transparent to the file and directory structure. It provides a framework so that different format may co-exist on the same media. This part of the standard is the most abstract and dry to read. It defines many terms that a UDF file system implementer rarely needs to care in file system operations.This part is most interesting for writers of the volume mount module (to identify that this is a UDF volume, i.e., the command mount_udf) and media formating module (to allow other system identify that this is a UDF volume,i.e., the command newfs_udf).To explain the volume structure, we step though the mount procedure to see how the UDF Volume is recognized. Mount always happens before the UDF media can be used by the host. This usually happens automatically when a removable UDF media is attached to the system, or it can be enabled manually by a command (say, the mount command on Unix type systems).The mount procedure can be separated into two parts: volume recognition andfile system verification.Volume recognition is the first step to make sure this is a UDF volume. It only tells that this is a UDF media but does not tell where the file system metadata. A quick format utility can simply erase the UDF volume by erasing the recognition sequence of the volume. The volume recognition procedure looks for the Volume Recognition Sequence (VRS) from a base address (UDF's term is Volume Recognition Space). For most media, the base address is the start of the media. For multi-session optical media (CD-R, DVD-R, DVD+R, BD-R etc), the base point is the start of the last data session. VRS consists of the following three contiguous sectors which are stored after the first 32KB of the base address: Beginning Extended Area Descriptor (BEA)Volume Sequence Descriptor (VSD) with id "NSR02" or "NSR03"Terminating Extended Area Descriptor (TEA)After volume recognition, the mounter must find the metadata of UDF to make sure this UDF volume is valid and its revision can be handled by the system. UDF metadata structures are called Descriptors in the standard. The start address of all descriptor are sector-aligned. Most descriptors are smaller than a sector (the bitmap and sparing table are two exceptions). Some descriptors contain pointers (i.e., addresses) of other descriptors. These descriptors are chained together in a certain order. A mounter may perform the following stepsto make sure the UDF media is mountable:Anchor Volume Descriptor Pointer (AVDP): find the AVDP at sector address 256. AVDP contains the start address and size of the main VolumeDescriptor Sequence (VDS) and a reserved VDS. The reserved VDS duplicate the data in the main VDS to increase the reliability. Only the main VDSneeds to be read by the mounter.VDS contains many descriptors until it is terminated by a TerminatingDescriptor (TD). The following descriptors are the most crucial to themounter: Partition Descriptor (PD) and Logical Volume Descriptor (LVD).Partition Descriptor (PD): PD states the start and size of apartition. All files and directories are stored in the partition.Most UDF media has PD. UDF also allows two PDs, one describes aread-only partition, and the other one describes an overwritepartition.Logical Volume Descriptor (LVD): LVD specifies the name of the volumethrough Logical Volume Identifier, defines all the physical andlogical partitions through Partition Map, and indicates the locationof the root directory through File Set. Partitions defines by thepartition map has a Partition Reference Number, i.e., the zero-basedindex of the partition in the partition map. Any sector in thepartition can be addressed by the Partition Reference Number and alogical address within the partition. UDF supports many differenttypes of partitions. The details will be discussed in section 5.5.The Integrity Sequence Extent of LVD contains the address of theLogical Volume Integrity Descriptor (LVID). LVID records the lasttime the media is written. The existence of a LVID tells that the UDFfile system is in a consistent state. An exception is that onwrite-once media using virtual partition, the write of the VirtualAllocation Table (VAT) File Entry (FE) replaces the function of LVID.Some OS (e.g., Mac OS X) requires a valid root directory when a filesystem is mounted. So the mounter could optionally verify that the rootdirectory is valid before mounting the file system. The partition mapdefined in LVD has enough information about the partitions on the media.The file set in LVD tells the address of the root directory FE. Themounter could read this FE to make sure the volume is valid.5.3 UDF Partition StructureUDF defines five different types of partitions. A partition provides a uniform interface to the file system layer while hiding the different underlying physical properties. Each partition has a partition reference number, which is the zero-based index in the Partition Map of the LVD. Blocks in a partition can be addressed by a block number ranging from 0 to N-1, where N is the size of the partition. The size of a partition may not be fixed. It may increase (for Virtual Partition, Metadata Partition, and Pseudo-Overwrite Partition) or decrease (for Metadata Partition).5.3.1 Type 1 PartitionThis is the simplest partition. A type 1 partition has a start address S and size N. A logical block number A in the partition can be converted to the media physical address (in UDF's term, the logical sector address) S+A. In certainoptical media, the start and size of the partition must be aligned to the packet size (such as 32KB). These special requirements are defined in the appendixes of the UDF standard. Free space of the partition is managed by the Unallocated Space Bitmap Descriptor. It contains one bit for each block of the partition. If the bit is set (1), the corresponding block is free. If it is clear (0), the corresponding block is allocated. The is contrary to whatFFS/UFS uses the bitmap, because the bitmap in UDF is called Unallocated Space Bitmap.5.3.2 Sparable PartitionSparable partitions are used on overwrite media that will fail after a certain number of overwrites (several thousands), such as CD-RW. In a file system, the places that are overwritten frequently are often important metadata area, e.g., bitmaps. Sparable partition allows the failed area to be remapped to other good part on the media so the failed area appears good to the upper level.A sparable partition is similar to a type 1 partition in the sense that it has a start address and size. In addition, it defines 2 to 4 sparing tables which points to reserved spare area on the media. Each sparing table points to the same reserved spare area. If one sparing table fails, another sparing table can be used instead. The unit of overwrite on such media is packet. For example, the packet size for CD-RW is 32 2K-sectors. One sector in packet failing means the whole packet fails. When this happens, the content of this packet iswritten to a spare area, and its new address is written to the sparing table. When translating a logical address in the sparable partition to the physical address, the sparing table is always consulted. If the logical address is not found in the sparing table, the address translation is the same as a type 1 partition. Otherwise, its new address in the sparing area recorded in the sparing table is returned. Thus, the sparing table acts as an exception table in the address translation. This mechanism guarantees that the logical address does not change when its original packet fails.We use an example to explain how the sparing table and address translation works. To make it more intuitive, we assume the packet size be 10 sectors, although in real optical media, the packet size is always a power of two. Assume the partition starts from physical address 1000 and has 8000 sectors. We have two spare areas starting from 500 and 9000, respectively. The size of each spare area is 50. Therefore, we have 5 packets in each spare area. Since each sparing table has the same content, we only show the content of the first sparing table. Before the media has any defects, the sparing table looks like below:Original Logical Address Mapped Physical Addressavailable500available510available520available530available540available9000available9010available9020available9030available9040UDF uses 0xFFFFFFFF to indicate that this spare packet is available. Since there is no defect, address translation is the same as a type 1 partition. So logical address 67 is translated to physical address 1067.Assume after some use, the system find the packet that contains block 93 fails when writing to it. It then write this packet to the spare packet with physical address 500, and update the sparing table:Original Logical Address Mapped Physical Address90500available510available520available530available540available9000available9010available9020available9030available9040Now the logical address translation is the same as before except for logical address 90-99. For example, logical address 67 still has the physical address 1067, but the logical address 97 now has physical address 507.As indicated in the example, the unit of sparing is a packet. The sparing table records the address of the first block of the packet.In this example, the spare area is outside of the partition. Actually, it can also be inside of the partition, and the partition then must mark the space occupied by the spare area unavailable for regular space allocation.For sparable partitions, the partition must start on a packet boundary, and its size must be an integral multiple of the packet size.5.3.3 Virtual PartitionVirtual partition is used on write-once media. Only three types of metadata are stored in the virtual partition: File Set Descriptor, File Entry (including Extended File Entry), and Allocation Extent Descriptor. If the file data is embedded in the file entry, these file data are also stored in the virtual partition. Virtual partition makes the write-once media appear as an overwrite media. Virtual partition layers on top of the type 1 partition. A Virtual Allocation Table (VAT) is used to map logical addresses of the virtualpartition to logical addresses in the underlying type 1 partition.We use a simple example to explain how it works. Assume the type 1 partitionstarts at physical block address 100. The File Set Descriptor has virtual address 0, and resides on physical block 100 (i.e., logical address 0), andFile Entry (FE) of the root directory has virtual address 1, and resides on physical block 101 (i.e., logical address 1). The VAT is an array of integers. To get the logical address of the virtual address x is VAT[x]. Currently VAT has two entries:1The above table tells that the virtual address is the same as the physical address currently. Assume the last written address on the media is 101. If an empty file foo is added to the root directory, the root directory FE is updated and written to physical address 102 (i.e., logical address 2), and the FE of file foo is written to physical address 103. Thus the VAT becomes:23The root directory FE still has virtual address 1, but now its logical address is 2. The virtual address of the FE of file foo is 2, and it is mapped to logical address 3, i.e., physical address 103. When the root directory is updated again, the VAT is changed again.VAT is stored as a special file in the type 1 partition. No other UDF data structures point to the VAT FE (called VAT ICB by UDF), and the latest VAT FE is always the last sector written on the media. When ejecting a media with virtual partition, the VAT and VAT FE are written after flushing all data. The UDF reader first finds the last written sector, and then it can get the VAT for the address translation.5.3.4 Metadata partitionMetadata partition is used to cluster metadata of the media together to get better performance. Metadata includes File Entries, allocation descriptors, directories, but does not include named streams or extended attributes.The metadata partition lies on top of the underlying partition, which could be a type 1 partition, sparable partition, or a pseudo-overwrite partition. The metadata partition consists of 3 files: the Metadata File, the Metadata Mirror File, and the Metadata Bitmap File. The Metadata File and Metadata Mirror File have duplicated metadata -- File Entries and Allocation Extent Descriptors. They may optionally have duplicated data, i.e., each metadata has two copies on the media. To simplify the following discussion, we assume that the Metadata Mirror File does not duplicate the Metadata File content.All data in the metadata partition are stored in the Metadata File. The logical block number in the metadata partition is the file offset in the Metadata File. Since some space in the Metadata File may be unused, the Metadata Bitmap File is used to keep track of the free space in the Metadata File. The metadata for the Metadata File, Metadata Mirror File, and Metadata Bitmap File are stored onthe underlying type 1 (or sparable or pseudo-overwrite) partition. These are the only metadata that are not stored in the metadata partition. The data of the Metadata File and Metadata Mirror File must be aligned to the media ECC block size or packet size, whichever is bigger, and its size must be a multiple of the media ECC block size or packet size, whichever is bigger.We use an example to explain how the metadata partition works. We assume the ECC block size and packet size is 10, although UDF requires it to be larger than 32, and the size in real media is always a power of two. Assume the underlying partition is a type 1 partition, starts at physical address 1000 and has 8000 sectors. The content of the Metadata File (i.e., the metadata partition) has two extents: the first starts at logical address 100 and has 300 sectors, the second starts at 2000 and has 500 sectors. Therefore, the size of the metadata partition is 800 sectors. The logical address 5 in the metadata partition means a block offset 5 in the Metadata File, which is translated to logical address 105 in the type 1 partition, or physical address 1105 in the physical media. We put more examples of address translation in the following table.Metadata Partition Logical Address Type 1 Partition LogicalAddressPhysical MediaAddress010011001342341234300200030007002400340075024503450The content of the Metadata Bitmap File is a Unallocated Space Bitmap Descriptor. Similar to the bitmap in a type 1 or sparable partition, the bitmap has one bit for each block in the partition.5.3.5 Pseudo-overwrite partitionThe pseudo-overwrite partition (POW) is used for next-generation write-once media (e.g., Blu-ray Disc recordable or BD-R) on next-generation intelligent drives. These drives manage the address translation within the drive (what the virtual partition does before) to make the partition appear as an overwritable although the physical media is write-once. When POW partition is used, the metadata partition shall also be used for metadata, in the hope that metadata are clustered and achieve better performance. However, on write-once media, even when data are logically clustered in one partition, they may physically be far apart on the media. Because a longer physical distance often implies poorer performance, whether the use of metadata partition can improve performance is questionable.In a media that supports POW partition, the media can be separated into several tracks. Each track has a Next Writable Address (NWA). A new block can bewritten to the NWA of any track. An existing block can be overwritten. The NWA of any track can change at any time. So NWA must be queried before any new block is written.5.3.6 Partition Descriptor and Partition Map。