Everything you ever wanted to know about Mac filesystems

What is AutoOptimize and AutoCluster?

Most people don't know about it, but since Panther (Mac OS 10.3) there has been a feature in the Mac OS that automatically defragments drives. When a file is opened, and it is fragmented into more than eight parts, the Mac OS automatically defragments the file by moving it to a new location. This process happens only if Journaling is turned on, and only on HFS+ volumes. This is called AutoOptimize. In my experience, installing Panther Enables journaling, so if you do not want AutoOptimize to be activated, switch off journaling in the Disk Utility. (Journaling is primarily used to restore unattended servers to operation in case they crash and there is major loss of data.)

AutoCluster (adaptive hot file clustering) is the second type of optimization done by the Mac OS. Over a period of 60 hours, the file system keeps track of files that are read frequently (for a file to be considered as a hot-file, it must be less than 10MB and never written to). At the end of this period, the "hottest" files (ie. the files that have been read the most times) are moved to the "hotband" of the disk (which is that part of the disk which is particularly fast given the physical characteristics of the disk).

The size of the "hotband" will depend on the size of the disk (ie. 5MB of hotband space for each GB of disk). "Cold" files that were in the hotband will be moved out of the hotband to make room for the hot files. As a side effect of being moved into the hotband, the hot files are defragmented. Currently, Adaptive Hot File Clustering only works on the boot volume, and only for Journaled HFS+ volumes that are more than 10GB.

The existence of these features has lead Apple to dispel the myth that "defragmentation is not necessary under OS X". However, exactly the opposite is true. Defragmentation is even more critical under OS X than is was under OS 9, because OS X keeps giant, multi-gigabyte virtual memory files (a feature that - unlike in OS 9 - cannot be turned off). These files get created and erased, and their contents changed, so often that they creates huge holes in the file system. Besides this, there are also swap files, which you can look at by choosing Go > Go To Folder in the finder and typing

/private/var/vm

Defragmentation is ever more necessary if you work with huge video or Photoshop files. Defragging can reduce wear and tear on the drive heads and speed up reading large files.

It is recommended to use a utility like TechTool Pro, Drive10 (available at micromat.com) or Coriolis Systems' iDefrag to defragment your drives annually. Before defragmenting, be sure to make a backup of all your files, in case the power goes out or some other catastrophe occurs while defragging.

(Note: I do not work for MicroMat or Coriolis, and am not being paid to promote their products. If you don't feel like dishing out $100 or so, you can also erase your drive and copy all the files back to it from the backup. This procedure takes a bit longer than using TechTool, and isn't so convenient, but it gets the job done.)
One MicroMat employee has this to say about defragmenting:

"Disk optimizers began on the Macintosh in an effort to improve the performance of early hard drives. With today’s high-speed drives, the amount of time required to open a file that is in 60 pieces appears to the user to be only slightly greater than the amount of time required to open it if it is one piece. However, the optimizer is now seen to have a purpose more important than performance.

In addition to ensuring that HFS+ volumes have sufficient free contiguous disk space for the disk directory to grow, disk optimizers are useful because they simplify the disk directory, causing all of the nodes in the Extents B-Tree to be free rather than used. A simplified disk directory is easier to repair or rebuild. One symptom of an excessively complex disk directory is an error messsage from Disk First Aid that the “hash table is full.” The hash table is created in RAM by Disk First Aid as it attempts to rebuild the disk directory. It is not a file on the disk itself.

Should you ever require the services of a data recovery firm, please be advised that your bill will be proportional to how badly fragmented your disk is. File recovery is greatly simplified when the pieces (extents) of a file do not require being searched for individually by a person.

Always make and test a backup before running any disk optimizer. It is prudent to check the volume structures (disk directory) of the disk before running the optimizer, and to perform a surface scan to check for bad blocks before the optimizer begins to move around large amounts of data. A UPS device to ensure a steady supply of electricity for models other than iBooks and PowerBooks is highly recommended.

The claim that installations of Mac OS X on HFS+ volumes do not fragment is a myth believed by people who do not have disk optimizers that allow them to see how much fragmentation their disks have. It is an example of ignorance that is not able to be removed by any amount of evidence. I think theologians call that 'invincible ignorance.' It is now a widespread form of the pollution of information space."

HFS+ volumes support colossal-sized files, which are generated when working with HD, SD, or DV video. This is much better than the Windows' filesystem, which limits files to 4 GB. All Windows programs need a "work around" that breaks all video into 4 GB bits and lines them up within the program, so it appears to the user as one file.

What is UFS?

Apple does not recommend using UFS disks unless you are a developer working on software for UNIX platforms. Apple’s implementation of UFS is said to be relatively slow, and if fsck cannot fix a disk directory error, there is no alternative to reformating the volume and restoring the files from a backup.

UFS disks attempt to reduce fragmentation by putting all the pieces of a file (still called “extents”) on the same cylinder. If the cylinder becomes full and new pieces must be added to the files on it, the files fragment. UFS disks use structures call indirect and double-indirect inodes to keep track of the locations of pieces of fragmented files.