Defragmentation
Defragmentation is advantageous and relevant to file systems on electromechanical
Causes of fragmentation
Fragmentation occurs when the file system cannot or will not allocate enough contiguous space to store a complete file as a unit, but instead puts parts of it in gaps between existing files (usually those gaps exist because they formerly held a file that the file system has subsequently deleted or because the file system allocated excess space for the file in the first place). Files that are often appended to (as with log files) as well as the frequent adding and deleting of files (as with emails and web browser cache), larger files (as with videos) and greater numbers of files contribute to fragmentation and consequent performance loss. Defragmentation attempts to alleviate these problems.
Example
An otherwise blank disk has five files, A through E, each using 10 blocks of space (for this section, a block is an allocation unit of the
However, if file F then needs to be expanded, there are three options, since the space immediately following it is no longer available:
- Move the file F to where it can be created as one contiguous file of the new, larger size. This would not be possible if the file is larger than the largest contiguous space available. The file could also be so large that the operation would take an undesirably long period of time.
- Move all the files after F until one opens enough space to make it contiguous again. This presents the same problem as in the previous example: if there are a small number of files or not much data to move, it isn't a big problem, but if there are thousands or even tens of thousands of files, there isn't enough time to move all those files.
- Add a new block somewhere else, and indicate that F has a second extent (see example 5 in the image). Repeat this hundreds of times and the filesystem will have a number of small free segments scattered in many places, and some files will have multiple extents. When a file has many extents like this, access time for that file may become excessively long because of all the random seeking the disk will have to do when reading it.
Additionally, the concept of “fragmentation” is not only limited to individual files that have multiple extents on the disk. For instance, a group of files normally read in a particular sequence (like files accessed by a program when it is loading, which can include certain DLLs, various resource files, the audio/visual media files in a game) can be considered fragmented if they are not in sequential load-order on the disk, even if these individual files are not fragmented; the read/write heads will have to seek these (non-fragmented) files randomly to access them in sequence. Some groups of files may have been originally installed in the correct sequence, but drift apart with time as certain files within the group are deleted. Updates are a common cause of this, because in order to update a file, most updaters usually delete the old file first, and then write a new, updated one in its place. However, most filesystems do not write the new file in the same physical place on the disk. This allows unrelated files to fill in the empty spaces left behind.
Mitigation
Defragmentation is the operation of moving file extents (physical allocation blocks) so they eventually merge, preferably into one. Doing so usually requires at least two copy operations: one to move the blocks into some free scratch space on the disk so more movement can happen, and another to finally move the blocks into their intended place. In such a paradigm, no data is ever removed from the disk, so that the operation can be safely stopped even in the event of a power loss. The article picture depicts an example.
To defragment a disk, defragmentation software (also known as a "defragmenter") can only move files around within the free space available. This is an intensive operation and cannot be performed on a filesystem with little or no free space. During defragmentation, system performance will be degraded, and it is best to leave the computer alone during the process so that the defragmenter does not get confused by unexpected changes to the filesystem. Depending on the algorithm used it may or may not be advantageous to perform multiple passes. The reorganization involved in defragmentation does not change logical location of the files (defined as their location within the
Besides defragmenting program files, the defragmenting tool can also reduce the time it takes to load programs and open files. For example, the Windows 9x defragmenter included the Intel Application Launch Accelerator which optimized programs on the disk by placing the defragmented program files and their dependencies next to each other, in the order in which the program loads them, to load these programs faster.[1] In Windows, a good defragmenter will read the Prefetch files to identify as many of these file groups as possible and place the files within them in access sequence.
At the beginning of the hard drive, the outer tracks have a higher data transfer rate than the inner tracks. Placing frequently accessed files onto the outer tracks increases performance.[2] Third party defragmenters, such as MyDefrag, will move frequently accessed files onto the outer tracks and defragment these files.[3]
Improvements in modern hard drives such as
Partitioning
A common strategy to optimize defragmentation and to reduce the impact of fragmentation is to
Offline defragmentation
The presence of immovable system files, especially a
In NTFS, as files are added to the disk, the
Solid-state disks
When reading data from a conventional electromechanical hard disk drive, the disk controller must first position the head, relatively slowly, to the track where a given fragment resides, and then wait while the disk platter rotates until the fragment reaches the head. A solid-state drive (SSD) is based on flash memory with no moving parts, so random access of a file fragment on flash memory does not suffer this delay, making defragmentation to optimize access speed unnecessary. Furthermore, since flash memory can be written to only a limited number of times before it fails, defragmentation is actually detrimental (except in the mitigation of catastrophic failure). However, Windows still defragments a SSD automatically (albeit less vigorously) to prevent the file system from reaching its maximum fragmentation tolerance (when the metadata can’t represent any more file fragments). Once the maximum fragmentation limit is reached, subsequent attempts to write to disk fail.[6]
SMR hard disks
Although many SMR hard disks will accept TRIM command, SMR hard disks still need defragmentation for improved performance.[citation needed]
Approach and defragmenters by file-system type
- Symantec Corporation, and the version that came with Windows 2000 and XP is licensed from Condusiv Technologies.
- defragmentation tool based on Diskeeper[9] that made use of the defragmentation APIs and was a snap-in for Computer Management. In Windows Vista, Windows 7 and Windows 8, the tool has been greatly improved and was given a new interface with no visual diskmap and is no longer part of Computer Management.[10][11] There are also a number of free and commercial third-party defragmentation products available for Microsoft Windows.
- This effectively controls system degradation after extended use.
- Btrfs has online and automatic defragmentation available.[13]
- Linux ext2, ext3, and ext4: Much like UFS, these filesystems employ allocation techniques designed to keep fragmentation under control at all times.[14] As a result, defragmentation is not needed in the vast majority of cases.[15] ext2 uses an offline defragmenter called
e2defrag
, which does not work with its successor ext3. However, other programs, or filesystem-independent ones such as defragfs,[16] may be used to defragment an ext3 filesystem. ext4 is somewhat backward compatible with ext3, and thus has generally the same amount of support from defragmentation programs. Currently e4defrag can be used to defragment an ext4 filesystem, including online defragmentation. - VxFS has theutility that includes defrag operations.
fsadm
- Mac OS 8.1 has a number of optimizations to the allocation algorithms in an attempt to defragment files while they are being accessed without a separate defragmenter.[18]There are several restrictions for files to be candidates for 'on-the-fly' defragmentation (including a maximum size 20MB). There is a utility, iDefrag, by Coriolis Systems available since OS X 10.3. On traditional Mac OS defragmentation can be done by Norton SpeedDisk and TechTool Pro.
- WAFL in NetApp's ONTAP 7.2 operating system has a command called
reallocate
that is designed to defragment large files. - XFS provides an online defragmentation utility called
xfs_fsr
. - SFS processes the defragmentation feature in almost completely stateless way (apart from the location it is working on), so defragmentation can be stopped and started instantly.[19]
- ADFS, the file system used by RISC OS and earlier Acorn Computers, keeps file fragmentation under control without requiring manual defragmentation.[20]
See also
- Comparison of defragmentation software
- Fragmentation (computing)
- File system fragmentation
- Virtual disk image
- Wear leveling, a similar technique for prolonging flash memory content
Notes
- ^ The practice of marking the now unused space of a deleted file in a table as available for later use (without erasing its contents), is why undelete programs are able to work; they recover files whose names have been deleted from the directory, but whose space has not yet been reused.[citation needed]
References
- ^ Cwdixon.com Archived 2010-10-06 at the Wayback Machine. Cwdixon.com. Retrieved on 2013-07-28.
- ^ The Ultimate Defragger - LaRud's Place. Larud.net (2012-01-19). Retrieved on 2013-07-28.
- ^ "MyDefrag v4.2.8". Archived from the original on 2010-02-16. Retrieved 2014-08-14.
On most harddisks the beginning of the harddisk is considerably faster than the end, sometimes by as much as 200 percent! You can measure this yourself with utilities such as * HD Tune. MyDefrag is therefore geared towards moving all files to the beginning of the disk.
- ^ Serdar Yegulalp (20 September 2005). "New hard disk drives reduce need for disk defragmentation". SearchWindowsServer.com: Disk Defragmentation Fast Guide. Archived from the original on 3 June 2008. Retrieved 2008-12-27.
- ^ "Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS -- MSDN Magazine, December 2001". Archived from the original on 2003-04-24. Retrieved 2006-12-19. msdn.microsoft.com: "The other big enhancement [in windows XP] is support for online defragmentation of the MFT and most directory and file metadata"
- ^ Hanselman, Scott (3 December 2014). "The real and complete story - Does Windows defragment your SSD?". Scott Hanselman's blog. Microsoft. Archived from the original on 22 December 2014.
- ^ Norton, Peter (October 1994). Peter Norton's Complete Guide to DOS 6.22. Sams. p. 521.
- ^ M. Kozierok, Charles (2001-04-17). "NTFS Versions". PC Guide. Archived from the original on 2015-09-24. Retrieved 2015-02-20.
- ^ Third-party disk defragmenter tools for Windows Archived 2011-11-28 at the Wayback Machine. Support.microsoft.com (2011-08-23). Retrieved on 2013-07-28.
- ^ "Disk Defragmentation – Background and Engineering the Windows 7 Improvements". Archived from the original on 2014-06-13. Retrieved 2014-06-15.
- ^ "New Defrag options in Windows 8". 13 November 2011. Archived from the original on 2015-02-20. Retrieved 2014-06-15.
- ^ "FreeBSD Man Pages". The FreeBSD Project. Archived from the original on 21 February 2015. Retrieved 21 February 2015.
- ^ "Linux kernel 3.0, Section 1.1. Btrfs: Automatic defragmentation, scrubbing, performance improvements". kernelnewbies.org. 2011-07-21. Archived from the original on 2016-03-30. Retrieved 2016-04-05.
- ^ "HTG Explains: Why Linux Doesn't Need Defragmenting". How-To Geek. Archived from the original on 2013-07-19. Retrieved 2013-08-01.
- ^ 5.10. Filesystems Archived 2013-05-27 at the Wayback Machine. Tldp.org (2002-11-09). Retrieved on 2013-06-22.
- ^ Erik Bärwaldt: Optimizing data organization on disk Archived 2014-09-06 at the Wayback Machine
- ^ "Journaling File System Support". eComStation. Archived from the original on 2008-12-08. Retrieved 2008-12-27.
- ^ "Fragmentation in HFS Plus Volumes". Archived from the original on 18 November 2012. Retrieved 2 September 2020.
As we have seen, an HFS+ volume seems to resist fragmentation rather well on Mac OS X 10.3.x, and I don't envision fragmentation to be a problem bad enough to require proactive remedies (such as a defragmenting tool).
- ^ "Detecting a file fragmentation point for reconstructing fragmented files using sequential hypothesis testing". US8407192 B2. Archived from the original on 21 February 2015. Retrieved 21 February 2015.
- ^ Reeves, Nick (26 October 1990). "E format design document". Archived from the original on 7 April 2013. Retrieved 24 May 2013.
Sources
- Norton, Peter (1994) Peter Norton's Complete Guide to DOS 6.22, page 521 – Sams (ISBN 067230614X)
- Woody Leonhard, Justin Leonhard (2005) Windows XP Timesaving Techniques For Dummies, Second Edition page 456 – For Dummies (ISBN 0-764578-839).
- Jensen, Craig (1994). Fragmentation: The Condition, the Cause, the Cure. Executive Software International (ISBN 0-9640049-0-9).
- ISBN 1-597490-792)
- Robb, Drew (2003) Server Disk Management in a Windows Environment Chapter 7 – AUERBACH (ISBN 0849324327)
External links
- The Big Windows 7 Defragmenter Test Benchmarks of popular defrag utilities
- Microsoft Windows XP defragmentation - How to schedule a weekly defragmentationMicrosoft Windows 2000 Professional and Server defragmentation - How to schedule defragmentation
- SST Hard Disk Optimizer
- How Linux avoids making files fragmented Archived 2016-06-04 at the Wayback Machine
- How defragmentation was changed for Windows 7
- Complete list of Defragmentation Utilities for Windows Archived 2016-05-28 at the Wayback Machine
- Does Your SSD's File System Affect Performance?