Maintaining your XFS with XFS Filesystem Reorganiser xfs_fsr to defrag
File Systems are a hairy topic, on Windows you should be using NTFS (the days of FAT are long gone!) but on Linux, BSD and *Solaris we still have a wide variety to pick and choose depending on our needs. I’ve always been a JFS and XFS fan (previously ReiserFS) until Btrfs goes mainstream (which is one thing to hangout for in Linux Kernel 2.6.29!) and often I’d have a mixture of all three. Our main server at home – affectionately dubbed Zeus, after our lovable Australian Customs puppy Zeus, uses XFS, JFS and Ext3.
JFS to manage the home directories and core file system, ReiserFS for the temp folder and XFS for the heavy file shares – which span multiple terrabytes of files over a LVM (with each file being 1-2Gb in size). The reasoning behind opting for XFS over another file system for the file server was that XFS performs incredibly well under heavy load and scales well when you know the files are big (over 500Mb). Overall I’ve always felt that XFS does provide consistent performance and scalabilty in comparison to the others – but you may think otherwise.
Unfortunately, XFS – whilst quite an excellent file system for managing large files, it seems, suffers from fragmentation over time (especially true if you use your file system for DVR – eg, a Myth backend host) or if the disk gets close to filling up. Luckily there are two utilities that XFS has to manage this fragmentation.
xfs_db– XFS Debug Information
Used to examine an XFS filesystem for problems or gather information about the XFS file system.xfs_fsr– File System Organiser
Improves the organisation of mounted file systems. The reorganisation algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).
In Debian/Ubuntu (and derivatives) these two utilities are found in the package xfsdump. Using these two utilities we can workout the health of the file system (xfs_db) and hopefully tune/optimise it (xfs_fsr). I took the plunge last night and optimised Zeus’s main file storage partition:
Filesystem Size Used Avail Use% Mounted on
/dev/sdf7 40G 3.5G 37G 9% /
varrun 1014M 4.5M 1010M 1% /var/run
varlock 1014M 8.0K 1014M 1% /var/lock
udev 1014M 112K 1014M 1% /dev
devshm 1014M 0 1014M 0% /dev/shm
lrm 1014M 34M 980M 4% /lib/modules/2.6.22-15-generic/volatile
/dev/sdf6 1023M 38M 986M 4% /boot
/dev/sdf10 235G 173G 63G 74% /home
/dev/sdf9 10G 544K 10G 1% /opt
/dev/sdf8 10G 2.7G 7.4G 27% /var
/dev/mapper/Storage
2.3T 1.9T 408G 83% /media/LVM/Storage
/dev/sde1 466G 396G 71G 85% /media/Backups
As you can see, the LVM “Storage” mount has just under 20% free space and the non-LVM partition for “Backups” has 15% free space. Both these are XFS volumes, to find the health of the two use the xfs_db command to gather some information.
$ sudo xfs_db -c frag -r /dev/mapper/Storage $ sudo xfs_db -c frag -r /dev/sde1
Here we’re asking xfs_db to open the file system in a readonly mode (-r) passing in a command (-c) to get the file fragementation data (frag) for the device (/dev/*). When we use the frag command, it returns information only pertaining to the file data in the filesystem as opposed to the fragmentation of freespace (which we can guage with passing the freesp command). The output of the commands appear below for Zeus.
thushan@ZEUS:~$ sudo xfs_db -c frag -r /dev/sde1 actual 189356, ideal 148090, fragmentation factor 21.79% thushan@ZEUS:~$ sudo xfs_db -c frag -r /dev/mapper/Storage actual 406056, ideal 21584, fragmentation factor 94.68%
Wow! The LVM partition (which spans 4 drives) has around 95% fragementation! Yikes!!! The parition has quite a few Virtual Machine images, various large files (DV Captures etc). The “Backup” (sde1) on the other hand isnt as badly fragmented.
So right now we’ve found our problem and its time to fix it. First thing to do – and realise that we can fix this on a live running system – is to try and find a time where the partition will be used very little (like overnight) so you let its do its thing without unnecessary burden. Then lets make use of the File System Organiser utility (xfs_fsr) and ask it to reorganise our parition to the best of its ability.
$ sudo xfs_fsr -t 25200 /dev/mapper/Storage -v $ sudo xfs_fsr -t 25200 /dev/sde1 -v
Now this is much simpler, the xfs_fsr utility is being told to reorganise /dev/* with a timeout (-t) of 7hrs (60 * 60 * 7 = 25200) which is specified in seconds. Because I like to see how much is done I also specified the verbose output option (-v). Let it do its thing and hopefully when you return you will have the last bit of output showing the extents before, how many after and the inode, something like this:
extents before:5 after:1 DONE ino=4209066103 ino=4209066107 extents before:5 after:1 DONE ino=4209066107 ino=4209066101 extents before:4 after:1 DONE ino=4209066101 ino=4209066091 extents before:3 after:1 DONE ino=4209066091 ino=4209066093 extents before:3 after:1 DONE ino=4209066093 ino=4209066105 extents before:2 after:1 DONE ino=4209066105 ino=4209066143 extents before:27 after:1 DONE ino=4209066143
Now its time to go back and check how well the file system reorganising was:
$ sudo xfs_db -c frag -r /dev/mapper/Storage
And the results?
thushan@ZEUS:~$ sudo xfs_db -c frag -r /dev/mapper/Storage actual 21652, ideal 21584, fragmentation factor 0.31%
Lovely! What a difference and you’ll notice the improvement immediately if you start moving or transfering files around.
Ideally, you may want to setup a cron task to let this process run (maybe with a lower timeout) overnight or when theres low-load. Whats great about the xfs_fsr utility is that its smart enough to remember where it finished up last time and continue from there. Its a shame Ubuntu doesnt do this already.
Believe it or not I actually stayed up waiting for it to hit the servers, then hopped over to the 
