Jump to content

Ext3: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
Jdowland (talk | contribs)
1,091 edits
→Defragmentation: lots of CNs for the subjective last para
Line 61: Line 61:
That being said, as the Linux System Administrator Guide states, "Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system."<ref>http://www.tldp.org/LDP/sag/html/filesystems.html</ref>
That being said, as the Linux System Administrator Guide states, "Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system."<ref>http://www.tldp.org/LDP/sag/html/filesystems.html</ref>


Irrespective of the above (subjective) statement, file fragmentation can be an important issue in server environments such as in multi-media server applications. While it is true that ext3 is more resistant to file fragmentation than FAT, and NTFS filesystems, nonetheless ext3 filesystems can and do get fragmented over time.<ref>"We found heavily fragmented free areas on an intensively used IMAP server which stores all its emails in individual files - although more than 900 GB of the total disk space of 1.4 TB were still available." http://www.heise-online.co.uk/open/Tuning-the-Linux-file-system-Ext3--/features/110398/3</ref> Consequently the successor to the ext3 filesystem, [[ext4]], includes a filesystem defragmentation utility and support for extents (contiguous file regions).
Irrespective of the above (subjective) statement, file fragmentation can be an important issue in server environments such as in multi-media server applications{{cn}}. While it is true that ext3 is more resistant to file fragmentation than FAT, and NTFS filesystems, nonetheless ext3 filesystems can and do get fragmented over time.<ref>"We found heavily fragmented free areas on an intensively used IMAP server which stores all its emails in individual files - although more than 900 GB of the total disk space of 1.4 TB were still available." http://www.heise-online.co.uk/open/Tuning-the-Linux-file-system-Ext3--/features/110398/3</ref> Consequently the successor to the ext3 filesystem, [[ext4]], includes a filesystem defragmentation utility and support for extents (contiguous file regions).


Further examples in which lack of defragmentation in some Linux filesystems (such as ext3) is a serious issue, includes server applications where rapid, concurrent and random file creation, update or access occurs. Such systems include large-scale carrier grade voice mail systems, Media-Messaging Service Centers (MMSCs) and SMS/SMSCs (Short Message Service Centers) servers. Media servers such as large scale voice mail and UMS servers are required to stream hundreds of voice or video streams concurrently to hundreds of users in near real-time conditions. These types of applications are particularly susceptible to file fragmentation; access delays during playback of a voice (e.g. voice mail) or video file, due to multiple fragmentation in the media file, can lead to playback interruption or distortion. As fragmentation increases over time, service capacity of these systems degrades because of increased CPU and I/O overhead resulting from fragmentation induced disk thrashing.
Further examples in which lack of defragmentation in some Linux filesystems (such as ext3) is a serious issue{{cn}}, includes server applications where rapid, concurrent and random file creation, update or access occurs. Such systems include large-scale carrier grade voice mail systems{{cn}}, Media-Messaging Service Centers{{cn}} (MMSCs) and SMS/SMSCs{{cn}} (Short Message Service Centers) servers. Media servers such as large scale voice mail and UMS servers are required to stream hundreds of voice or video streams concurrently to hundreds of users in near real-time conditions. These types of applications are particularly susceptible to file fragmentation{{cn}}; access delays during playback of a voice (e.g. voice mail) or video file, due to multiple fragmentation in the media file, can lead to playback interruption or distortion. As fragmentation increases over time, service capacity of these systems degrades because of increased CPU and I/O overhead resulting from fragmentation induced disk thrashing.


=== Compression ===
=== Compression ===

Revision as of 22:27, 9 December 2008

ext3
Developer(s)Stephen Tweedie
Full nameThird extended file system
IntroducedNovember 2001 with Linux 2.4.15
Partition IDs0x83 (MBR)
EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (GPT)
Structures
Directory contentsTable, hashed B-tree with dir_index enabled
File allocationbitmap (free space), table (metadata)
Bad blocksTable
Limits
Max volume size2 TiB – 16 TiB
Max file size16 GiB – 2 TiB
Max no. of filesVariable, allocated at creation time[1]
Max filename length255 bytes
Allowed filename
characters		All bytes except NUL and '/'
Features
Dates recordedmodification (mtime), attribute modification (ctime), access (atime)
Date rangeDecember 14, 1901 - January 18, 2038
Date resolution1s Nanosecond (using undocumented big i-node)
AttributesNo-atime, append-only, synchronous-write, no-dump, h-tree (directory), immutable, journal, secure-delete, top (directory), allow-undelete
File system
permissions		Unix permissions, ACLs and arbitrary security attributes (Linux 2.6 and later)
Transparent
compression		No
Transparent
encryption		No (provided at the block device level)
Data deduplicationNo
Other
Supported
operating systems		Linux, BSD, Windows (through an IFS)

The ext3 or third extended filesystem is a journaled file system that is commonly used by the Linux operating system. It is the default file system for many popular Linux distributions. Stephen Tweedie first revealed that he was working on extending ext2 in a February 1999 kernel mailing list posting[2] and the filesystem was merged with the mainline Linux kernel in November 2001 from 2.4.15 onward.[3] Its main advantage over ext2 is journalling which improves reliability and eliminates the need to check the file system after an unclean shutdown.

Advantages

Although its performance (speed) is less attractive than competing Linux filesystems such as JFS, ReiserFS and XFS, it does have the significant advantage that it does allow in-place upgrades from the ext2 file system without having to back up and restore data as well as using less CPU power than ReiserFS and XFS.[4] It is also considered safer than the other Linux file systems due to its relative simplicity and wider testing base.

The ext3 file system adds, over its predecessor:

Without these, any ext3 file system is also a valid ext2 file system. This has allowed well-tested and mature file system maintenance utilities for maintaining and repairing ext2 file systems to also be used with ext3 without major changes. The ext2 and ext3 file systems share the same standard set of utilities, e2fsprogs, which includes a fsck tool. The close relationship also makes conversion between the two file systems (both forward to ext3 and backward to ext2) straightforward.

While in some contexts the lack of "modern" filesystem features such as dynamic inode allocation and extents could be considered a disadvantage, in terms of recoverability this gives ext3 a significant advantage over file systems with those features. The file system metadata is all in fixed, well-known locations, and there is some redundancy inherent in the data structures that may allow ext2 and ext3 to be recoverable in the face of significant data corruption, where tree-based file systems may not be recoverable.

Journaling levels

There are three levels of journaling available in the Linux implementation of ext3:

Journal (lowest risk)
Both metadata and file contents are written to the journal before being committed to the main file system. Because the journal is relatively continuous on disk, this can improve performance in some circumstances. In other cases, performance gets worse because the data must be written twice - once to the journal, and once to the main part of the filesystem.
Ordered (medium risk)
Only metadata is journaled; file contents are not, but it's guaranteed that file contents are written to disk before associated metadata is marked as committed in the journal. This is the default on many Linux distributions. If there is a power outage or kernel panic while a file is being written or appended to, the journal will indicate the new file or appended data has not been "committed", so it will be purged by the cleanup process. (Thus appends and new files have the same level of integrity protection as the "journaled" level.) However, files being overwritten can be corrupted because the original version of the file is not stored. Thus it's possible to end up with a file in an intermediate state between new and old, without enough information to restore either one or the other (the new data never made it to disk completely, and the old data is not stored anywhere). Even worse, the intermediate state might intersperse old and new data, because the order of the write is left up to the disk's hardware.[5] [6]
Writeback (highest risk)
Only metadata is journaled; file contents are not. The contents might be written before or after the journal is updated. As a result, files modified right before a crash can become corrupted. For example, a file being appended to may be marked in the journal as being larger than it actually is, causing garbage at the end. Older versions of files could also appear unexpectedly after a journal recovery. The lack of synchronization between data and journal is faster in many cases. XFS and JFS use this level of journaling, but ensure that any "garbage" due to unwritten data is zeroed out on reboot.

Disadvantages

Functionality

Since ext3 aims to be backwards compatible with the earlier ext2, many of the on-disk structures are similar to those of ext2. Because of that, ext3 lacks a number of features of more recent designs, such as extents (ext3 does not use extents, a feature found in newer filesystem designs such as JFS, ext4 and XFS. Instead, a simpler indirect block mapping scheme is used which is less efficient with large files, in terms of storage and processing power.[7]), dynamic allocation of inodes, and block suballocation. There is a limit of 31998 sub-directories per one directory, stemming from its limit of 32000 links per inode.[8]

ext3, like most current Linux filesystems, cannot be fsck-ed while the filesystem is mounted for writing. Attempting to check a file system that is already mounted may detect bogus errors where changed data has not reached the disk yet, and corrupt the file system in an attempt to "fix" these errors.

Defragmentation

There is no online ext3 defragmentation tool that works on the filesystem level. An offline ext2 defragmenter, e2defrag, exists but requires that the ext3 filesystem be converted back to ext2 first. But depending on the feature bits turned on the filesystem, e2defrag may destroy data; it does not know how to treat many of the newer ext3 features.[9]

There are userspace defragmentation tools like Shake[10] and defrag [11]. Shake works by allocating space for the whole file bolt upright and hoping that it will make the newly allocated file less fragmented. It also tries to write files used at the same time next to each other. Defrag works by copying each file over itself. However they only work if the filesystem is reasonably empty. A true defragmentation tool does not exist for ext3.[12]

That being said, as the Linux System Administrator Guide states, "Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system."[13]

Irrespective of the above (subjective) statement, file fragmentation can be an important issue in server environments such as in multi-media server applications[citation needed]. While it is true that ext3 is more resistant to file fragmentation than FAT, and NTFS filesystems, nonetheless ext3 filesystems can and do get fragmented over time.[14] Consequently the successor to the ext3 filesystem, ext4, includes a filesystem defragmentation utility and support for extents (contiguous file regions).

Further examples in which lack of defragmentation in some Linux filesystems (such as ext3) is a serious issue[citation needed], includes server applications where rapid, concurrent and random file creation, update or access occurs. Such systems include large-scale carrier grade voice mail systems[citation needed], Media-Messaging Service Centers[citation needed] (MMSCs) and SMS/SMSCs[citation needed] (Short Message Service Centers) servers. Media servers such as large scale voice mail and UMS servers are required to stream hundreds of voice or video streams concurrently to hundreds of users in near real-time conditions. These types of applications are particularly susceptible to file fragmentation[citation needed]; access delays during playback of a voice (e.g. voice mail) or video file, due to multiple fragmentation in the media file, can lead to playback interruption or distortion. As fragmentation increases over time, service capacity of these systems degrades because of increased CPU and I/O overhead resulting from fragmentation induced disk thrashing.

Compression

Support for transparent compression is available as an unofficial patch for ext3. This patch is a direct port of e2compr and still needs further development, it compiles and boots well with upstream kernels but journaling is not implemented yet. The current patch is named e3compr and can be found here: http://sourceforge.net/projects/e3compr/

Size limits

ext3 has a maximum size for both individual files and the entire filesystem. For the filesystem as a whole that limit is 231−1 blocks. Both limits are dependent on the block size of the filesystem; the following chart summarizes the limits[15]:

Block size Max file size Max filesystem size
1KiB 16GiB <2TiB
2KiB 256GiB <4TiB
4KiB 2TiB <8TiB
8KiB[limits 1] 2TiB <16TiB
  1. ^ 8 KiB block size is only available on architectures which allow 8 KiB pages, such as Alpha.

No checksumming in journal

Ext3 does not do checksumming when writing to the journal. If barrier=1 is not enabled as a mount option (in /etc/fstab), and if the hardware is doing out-of-order write caching, one runs the risk of severe filesystem corruption during a crash.[16][17]

Consider the following scenario: If hard disk writes are done out-of-order (due to modern hard disks caching writes in order to amortize write speeds), it is likely that one will write a commit block of a transaction before the other relevant blocks are written. If a power failure or unrecoverable crash should occur before the other blocks get written, the system will have to be rebooted. Upon reboot, the file system will replay the log as normal, and replay the "winners" (transactions with a commit block, including the invalid transaction above which happened to be tagged with a valid commit block). The unfinished disk write above will thus proceed, but using corrupt journal data. The file system will thus mistakenly overwrite normal data with corrupt data while replaying the journal. There is a test program available to trigger the problematic behavior. If checksums had been used, where the blocks of the "fake winner" transaction were tagged with a mutual checksum, the file system could have known better and not replayed the corrupt data onto the disk. Journal checksumming has been added to EXT4.[18]

The ext3 barrier option is not enabled by default on almost all popular Linux distributions[citation needed], and thus most distributions are at risk. In addition, filesystems going through the device mapper interface (including software RAID and LVM implementations) may not support barriers, and will issue a warning if that mount option is used.[19][20] There are also some disks that do not properly implement the write cache flushing extension necessary for barriers to work, which causes a similar warning.[21] In these situations, where barriers are not supported or practical, reliable write ordering is possible by turning off the disk's write cache and using the data=journal mount option.[22]

ext4

Main article: ext4

An enhanced version of the filesystem is under development; it was announced by Theodore Ts'o [23] on June 28, 2006 under the name of ext4.

See also

References

  1. ^ The maximum number of inodes (and hence the maximum number of files and directories) is set when the file system is created. If V is the volume size in bytes, then the default number of inodes is given by V/213 (or the number of blocks, whichever is less), and the minimum by V/223. The default was deemed sufficient for most applications. The max number of subdirectories in one directory is fixed to 32000.
  2. ^ Stephen C. Tweedie (February 17 1999). "Re: fsync on large files". Linux kernel mailing list. {{cite web}}: Check date values in: |date= (help)
  3. ^ Rob Radez (November 23 2001). "2.4.15-final". Linux kernel mailing list. {{cite web}}: Check date values in: |date= (help)
  4. ^ Justin Piszcz. "Benchmarking Filesystems Part II". Linux Gazette (122).
  5. ^ curious onloooker: Speeding up ext3 filesystems
  6. ^ Common threads: Advanced filesystem implementor's guide, Part 8
  7. ^ Rob Radez (2005). "Extents,delayed allocation". future of ext3.
  8. ^ "How many sub-directories?".
  9. ^ Andreas Dilger. "Post to the ext3-users mailing list". ext3-users mailing list post.
  10. ^ Vleu.net: Shake
  11. ^ Index of /apps/defrag
  12. ^ RE: searching for ext3 defrag/file move program
  13. ^ http://www.tldp.org/LDP/sag/html/filesystems.html
  14. ^ "We found heavily fragmented free areas on an intensively used IMAP server which stores all its emails in individual files - although more than 900 GB of the total disk space of 1.4 TB were still available." http://www.heise-online.co.uk/open/Tuning-the-Linux-file-system-Ext3--/features/110398/3
  15. ^ Matthew Wilcox. "Documentation/filesystems/ext2.txt". Linux kernel source documentation.
  16. ^ Re: Frequent metadata corruption with ext3 + hard power-off
  17. ^ Re: Frequent metadata corruption with ext3 + hard power-off
  18. ^ ext4: Add the journal checksum feature
  19. ^ Re: write barrier over device mapper supported or not?
  20. ^ XFS and zeroed files
  21. ^ Barrier Sync
  22. ^ Re: Frequent metadata corruption with ext3 + hard power-off
  23. ^ LKML: "Theodore Ts'o": Proposal and plan for ext2/3 future development work
Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=Ext3&oldid=256922940"