Thushan Fernando Uncut

Now that I’ve decided what I want out of the server (and the hardware I’ve got), its time to workout what operating system to run the system on. Currently, ZEUS is running on Ubuntu Gutsy (7.10) which is running LVM with an XFS volume holding approximately 2.5Tb worth of data. There’s a cron job that defrags the XFS volume to keep things in order.

The Operating System

As the operating system is no longer maintained (my oversight into how long it would survive) I have to find an OS that supports the hardware platform without hacky hacky bits (and by this I mean avoiding buggy ACPI and issues with the NForce4 chipset and IRQ problems) and has a file system that will benefit long term.

There were a few considerations:

• Ubuntu 8.04.x LTS
  I like Ubuntu, I’m comfortable with the user land and find the Debian package system (in particular the dependency resolving) most impressive. Hardware is well supported and 8.04.3 (at the time of writing) boots on the hardware I originally selected (Intel) and the new configuration I recently selected (AMD). I could most definitely use Ext4 but the problems with data-loss (which I’ve reproduced on several occasions on desktop machines) scare me.FileSystem: I’d have to adopt either XFS or Ext4 on an LVM to factor in future-proofing, maybe get some fakeRAID happening for redundancy.
  Installation: comes with a Server edition that’s bare bones allowing it to be a minimalistic installation which is always nice!
• Ubuntu 9.04
  Initially when I started to rebuild Zeus back in April I wanted to use Ubuntu 9.04, I was really excited about Ext4 and the promise of a brand-spanking new file-system and what it would bring to the table. Unfortunately after using Ext4 with 9.04 I’ve come to realise its probably not the wisest to trust your data with it just yet – unless you get yourself a UPS! Laptop seems to be chugging nicely though.Installation: Like LTS, comes with a Server edition that’s bare bones allowing it to be a minimalistic installation which is always nice! (copy/paste!) Unfortunately picking 9.04 when 9.10 is just around the corner is not going to be ideal, I’ll be stuck with where I am right now in a year or so.

• CentOS 5.3 or wait for CentOS 5.4 coming soon!
  CentOS is based off Redhat Enterprise Linux and having only started using CentOS after joining HSD (though my first splash in Linux was with Bowtie Redhat Linux 6.0 – I still have the CD!) I find the RHEL clone to be stable and solid as a rock! CentOS 5.4 will (thanks to RHEL 5.4) bring some KVM goodness that is a high priority on my list.

  FileSystem: Ext4 is not in the RHEL5 stream (have to wait for RHEL6 – which is a good thing!) so the best bet here is to use XFS with LVM.
  Installation: Customised via kickstarts or setup a cobbler server to install via PXE.

• OpenSolaris
  OpenSolaris has some really exciting technologies that are worth a look at, but in particular is ZFS. ZFS breaks all the rules – and for a good reason. I’ve yet to really use Solaris – quite cosy with Linux, but there’s nothing like getting used to another OS by jumping straight into the deep end!

  FileSystem: ZFS all the way baby!
  Installation: Installation is via a LiveCD, I’ve looked into customising the installation so it removes some bits I don’t which I’ll discuss later but its quite fiddly.

So in case the sudden influx of OpenSolaris posts didnt give you the hint, I decided on OpenSolaris to power the new iZeus 2.0, actually no that sounds lame, zeusy will be the new ZEUS until ZEUS is retired in which case zeusy becomes zeus (confused?).

Why ZFS?

ZFS is one of those file-systems you look at and think, wow! Why didn’t anyone else think of that before?

• Very simple administration – you only use two commands, zpool and zfs.
• Highly scalable – 128-bit means we can hold 16 exabytes or 18 Million terabytes worth of data! More porn for you! XFS can no doubt handle the TBs we use for our home boxes now, but no-chance you can get the performance or benefits of ZFS in Ext3/Ext4 or XFS.
• Data integrity to heal a filesystem (no fsck’ing around!) – 256bit checksuming to protect data, if ZFS detects a problem it will attempt to reconstruct the bad block and continue on its merry way (utilising available redundancy)
• Compression – you can elect to compress a particular file-system or a hierarchy just by setting one command! I’m thinking things like logs here.
• No hardware dependency – JBOD on a controller, let ZFS maintain the RAID volumes in software. Checkout Michael Pryc’s crazy adventure with ZFS using USB thumb drives and Constantin’s original voyage with USB drives! RAID-Z is essentially RAID-5 without the write-hole problems has plagued it if power is lost during a write, it can also survive a loss of a drive (with RAIDZ-2 you can loose two drives).
• Happy snaps for free! Snapshot (a live) file-system as many times as you like, again one easy command. Its like that tendency to hit {CTRL+S} when your working in Windows from back in the days of Windows 9x, snapshot regularly!

So ZFS sounds much like marketing spiel right now, best thing since sliced bread, cooler than a cucumber, and you’d be right it is cool and the best thing since filesystems came to being. Over the coming days I’ll post some more on my musings with ZFS – keeping in mind that I’m still learning these things. It helps to have lots of hardware to play with, but even if you don’t, you can knock up a virtual version of OpenSolaris in VirtualBox, create some virtual disks and try it out.

There are a few caveats that I’ve come across though using ZFS, one is memory! ZFS will try and cache as much data as it can in RAM, so if you have 8Gb of RAM (as I have in this box) it will happily use as much of it as it can afford. Rightfully so, I was getting ~96MB/s transfering a 16Gb MPEG from one box to the other over our Gig link (thats from one end of the house to the other!) mind you this was just a test configuration using 2x 74Gb Western Digital Raptors (WD740ADFD) in a RAID-0 style hitting a single 150Gb Western Digital Raptor (WD1500ADFD). They could have gone much higher, but I was happy with that.

There are also (as of writing) no recovery tools for ZFS, but these are slated to arrive soon (Q4 2009) which is quite scary after you read this post about a guy loosing 10Tb worth of data, however a possible revert to an older uberblock may fix some problems.

Virtualisation

Initially I wanted to concentrate quite a bit on Virtualisation, I tried Xen on OpenSolaris. It was quite easy to setup a Xen Dom0 in OpenSolaris but with the 2009.06 release you had to tweak the Xen setup a bit. I wasn’t too enthusiastic about using Xen after seeing the performance lag in Windows in my musings. Instead I’m opting for my crush, VirtualBox.

So why use VirtualBox when you can get a bare-metal hypervisor? Firstly, performance seems to be sluggish with Xen for me (I didn’t investigate this too much), secondly I want to be able to run the latest and greatest OS’s out without worrying about upgrading Xen (I’m a sucker for OS’s!). VirtualBox development has accelerated at a feverish pace, I started with VirtualBox 1.3 in 2007 and its come an insanely long way since then. When a new release comes along, its as easy as updating VirtualBox and getting all the benefits. Plus with SunOracle‘s backing of VirtualBox you know things are going to work well on OpenSolaris, the Extras repository of VirtualBox makes it as easy as doing a pkg update.

I’m still quite intrigued by the way KVM is heading and how it will pan out, but for the future zeus, it will be VirtualBox.

I’ve been messing about with OpenSolaris (you’ll know why soon!) and decided to install the OpenSolaris Extras repository so I can grab the latest VirtualBox install from the repository. This repository has the following packages (as of writing) and is recommended if you plan on using VirtualBox:

NAME (PUBLISHER)                              VERSION         STATE      UFIX
SUNWadmj (extra)                              0.5.11-0.111    known      ----
SUNWjsnmp (extra)                             0.5.11-0.111    known      ----
SUNWwbapi (extra)                             0.5.11-0.111    known      ----
SUNWwbcou (extra)                             0.5.11-0.111    known      ----
SUNWwbdev (extra)                             0.5.11-0.111    known      ----
develop/java/javafx-sdk (extra)               1.2.0.233-0.111 known      ----
service/compute/sungridengine (extra)         6.2.2-0.111     known      ----
service/compute/sungridengine/arco (extra)    6.2.2-0.111     known      ----
service/compute/sungridengine/domainmanager (extra) 6.2.2-0.111     known      ----
system/font/truetype/ttf-fonts-core (extra)   1.0-0.111       known      ----
system/iiim/ja/atok (extra)                   17-0.111        known      ----
system/iiim/ja/wnn8 (extra)                   8-0.111         known      ----
virtualbox (extra)                            3.0.8-0.101     known      ----
virtualbox/kernel (extra)                     3.0.8-0.101     known      ----
web/firefox/plugin/flash (extra)              10.0.32.18-0.111 known      ----

So what do you need to get these freebies? (source help)

• Register if you haven’t already with Sun, otherwise login to your Sun Online Account get your certificates.
• Download the Key and Certificate files onto your desktop. They are named OpenSolaris_extras.key.pem and OpenSolaris_extras.certificate.pem respectively.
• Now we need to create a directory in /var/pkg to store the certificates – ensuring they have the correct permissions. Then we’ll add them to the folder.

  $ pfexec mkdir -m 0755 -p /var/pkg/ssl
  $ pfexec cp -i ~/Desktop/OpenSolaris_extras.key.pem /var/pkg/ssl
  $ pfexec cp -i ~/Desktop/OpenSolaris_extras.certificate.pem /var/pkg/ssl

• Then we add them to our configuration.

  $ pfexec pkg set-authority \
      -k /var/pkg/ssl/OpenSolaris_extras.key.pem \
      -c /var/pkg/ssl/OpenSolaris_extras.certificate.pem \
      -O https://pkg.sun.com/opensolaris/extra extra

• To test the above worked get a list of the packages in the repository with the command below. Your output should be similar to mine above.

  $ pkg list -a 'pkg://extra/*'

Now make sure your datetime settings are valid when you do the above, as I found mine was a little out of date and raised a few python exceptions.

After a bit of digging around, my original spec’d hardware I’ve decided is too much for a boxen that will be on 24×7, especially with the rates for electricity going up next year – every little Watt counts. The existing 65W CPU isn’t ideal, instead I’m opting for a 45W CPU instead and this means – looking at the lineup, its going to be a walk down AMD way. Less watts, less heat and less noise, noice! See AMD’s product roadmap for 2010-2011.

The original specifications I mentioned were:

• Motherboard: ASUS P5QL-PRO
• CPU: Intel Core-2 E6750 – 2.66Ghz (65W TDP, VT)
• RAM: Corsair TWIN2X4096-6400C5 (4Gb kit x 2 = 8Gb)
• Graphics: ASUS 9400GT PCI-Express

I’ve decided to change the CPU and Motherboard but keep the other bits and bobs – I could loose the graphics card and go onboard but I felt like leaving it there for now. The target budget is $250 maximum for both CPU+Mobo, so this means I’m sticking with DDR2 which implies AM2+ but it must also satisfy:

• CPU has to be 45W and be atleast 1.6Ghz, dual core no more, has to support Virtualization.
• Motherboard has to Support 8Gb (most boards doo!),  have atleast 2x  PCIe and a PCI slot, it would be nice if the network cards work (gigabit) but no fuss if it doesnt. No crazy shebangabang Wifi, remotes etc bling and if it has onboard Video great, otherwise its OK to use a crappy card.

I picked the AMD Athlon X2 5050e CPU because it was cheap (~$80), supports a 45W, has virtualisation and is an AM2. Next was the motherboard, looking at the ASUS, Gigabyte & XFX models as my target.

Chipset wise only the following fit the criteria for a possible match because others just don’t have the number of SATA ports available onboard. Primarily AMD boards are supplied by NVIDIA or AMD themselves.

• NVIDIA nForce 750a SLI
• NVIDIA GeForce 8200
• NVIDIA GeForce 8300
• AMD 790X Chipset

Initially I looked at the ASUS  boards (they’ve been nothing but rock solid for me in the past) but after a lot of research scouring through the manufacturer sites I ended up picking out the Gigabyte GA-MA790X-UD4P which is based on the AMD 790X Chipset. The board came with 8x SATA Ports, 3x PCIe and 2x PCI and a  Gigabit NIC all for a $137 from PCCaseGear. Not only was the power consumption lowered but the noise and heat generated was substantially lower too!

Coming in close was the ASUS M4N78 PRO or the ASUS M4A78 PRO, each of those unfortunately didn’t have as many SATA ports (2-less) nor the PCIe ports (1-less).

Sun Microsystems (pwnd by the Oracle) has release VirtualBox 2.2.4.

VirtualBox 2.2.4 (released 2009-05-29)

This is a maintenance release. The following items were ?xed and/or added:

• Windows Installer: ?xed a potential hang during installation
• Windows Installer: ?xed several problems (bug #3892)
• Solaris hosts: make it work with Solaris build 114 or later (bug #3981)
• Solaris hosts: ?xed a bug serial port character handling found during loopback (bug #3120)
• Linux hosts: adapted vboxdrv.sh to the latest changes in VBoxManage list runningvms (bug #4034)
• Windows hosts: ?xed a crash caused by host-only/bridged networking
• Mac OS X hosts: ?xed access to host DVD with passthrough disabled (bug #4077)
• Guest Additions: ?xed problems with KDE 4 not recognizing mouse clicks
• Windows Additions: ?xed incorrect 8-bit guest color depth in Windows 7 guests
• GUI: warn if VT-x/AMD-V could not be enabled for guests that require this setting (bug #4055)
• VMM: ?xed occassional crash due to insuf?ent memory
• VMM: ?xed hanging 64 bits Solaris guests
• VMM: restore from a saved state occassionally failed (bugs #3984 and #2742)
• Clipboard: ?xed a deadlock while shutting down the shared clipboard on X11 hosts (bug #4020)
• OVF: ?xed potential hang during import
• OVF: ?xed potential crashes during import/export on Win64 hosts
• VBoxManage modifyhd –compact: ?xed bug which could lead to crashes and image corruption (bug #3864)
• VBoxManage metrics collect: now ?ushes the output stream.
• VHD: made VBoxManage internalcommands sethduuid work for .vhd ?les (bug #3443)
• VHD: some .vhd ?les could not be cloned (bug #4080)
• VMDK: ?xed creating snapshots
• NAT: improvement of TCP connection establisment (bug #2987)
• NAT: ?xed order of DNS servers in DHCP lease (bug #4091)
• NAT: ?xed DHCP lease for multiple name servers (bug #3692)
• NAT: ?xed a potential segfault if the host lost its connectivity (bug #3964)
• Shared Folders: deny access to parent directories on Windows hosts (bug #4090)
• Shared Folders: make rm/rmdir work with Solaris guests on Windows hosts
• Networking: ?xed the problem with blocked receiving thread when a broadcast packet arrives too early to be handled by uninitialized e1000 adapter.
• Networking: ?xed the problem that caused host freezes/crashes when using bridged mode with host’s interface having RX checksum of?oading on (bug #3926 and related). Fixes problems with TX of?oading as well (bug #3870)
• PXE boot: Added support for PRO/1000 MT Server adapter.
• Python bindings: ?xed keyword con?ict
• SCSI: ?xed occasional crashes on Win64
• Serial: allow to redirect the serial port to a raw ?le (bug #1023)
• VRDP: ?xed a rare incorrect screen update

This release’s most significant update is the bugfix for truncating your files to 300Kb when attempting to compact the hard-disk – damn annoying! Update mos def.

Download:

• VirtualBox 2.2.4 for Windows hosts x86/amd64

• VirtualBox 2.2.4 for OS X hosts Intel Macs

• VirtualBox 2.2.4 for Linux hosts

• VirtualBox 2.2.4 for Solaris and OpenSolaris hosts x86/amd64

• VirtualBox 2.2.4 Software Developer Kit (SDK) All platforms (registration required)

Enjoy, oh btw did you know that OpenSolaris 2009.06 is coming tomorrow (Monday?)

File Systems are a hairy topic, on Windows you should be using NTFS (the days of FAT are long gone!) but on Linux, BSD and *Solaris we still have a wide variety to pick and choose depending on our needs. I’ve always been a JFS and XFS fan (previously ReiserFS) until Btrfs goes mainstream (which is one thing to hangout for in Linux Kernel 2.6.29!) and often I’d have a mixture of all three. Our main server at home – affectionately dubbed Zeus, after our lovable Australian Customs puppy Zeus, uses XFS, JFS and Ext3.

JFS to manage the home directories and core file system, ReiserFS for the temp folder and XFS for the heavy file shares – which span multiple terrabytes of files over a LVM (with each file being 1-2Gb in size). The reasoning behind opting for XFS over another file system for the file server was that XFS performs incredibly well under heavy load and scales well when you know the files are big (over 500Mb). Overall I’ve always felt that XFS does provide consistent performance and scalabilty in comparison to the others – but you may think otherwise.

Unfortunately, XFS – whilst quite an excellent file system for managing large files, it seems, suffers from fragmentation over time (especially true if you use your file system for DVR – eg, a Myth backend host) or if the disk gets close to filling up. Luckily there are two utilities that XFS has to manage this fragmentation.

• xfs_db – XFS Debug Information
  Used to examine an XFS filesystem for problems or gather information about the XFS file system.
• xfs_fsr – File System Organiser
  Improves the organisation of mounted file systems. The reorganisation algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).
  

In Debian/Ubuntu (and derivatives) these two utilities are found in the package xfsdump. Using these two utilities we can workout the health of the file system (xfs_db) and hopefully tune/optimise it (xfs_fsr). I took the plunge last night and optimised Zeus’s main file storage partition:

Filesystem            Size Used Avail Use% Mounted on
/dev/sdf7              40G  3.5G   37G   9% /
varrun               1014M  4.5M 1010M   1% /var/run
varlock              1014M  8.0K 1014M   1% /var/lock
udev                 1014M  112K 1014M   1% /dev
devshm               1014M     0 1014M   0% /dev/shm
lrm                  1014M   34M  980M   4% /lib/modules/2.6.22-15-generic/volatile
/dev/sdf6            1023M   38M  986M   4% /boot
/dev/sdf10            235G  173G   63G  74% /home
/dev/sdf9              10G  544K   10G   1% /opt
/dev/sdf8              10G  2.7G  7.4G  27% /var
/dev/mapper/Storage
                      2.3T  1.9T  408G  83% /media/LVM/Storage
/dev/sde1             466G  396G   71G  85% /media/Backups

As you can see, the LVM “Storage” mount has just under 20% free space and the non-LVM partition for “Backups” has 15% free space. Both these are XFS volumes, to find the health of the two use the xfs_db command to gather some information.

$ sudo  xfs_db -c frag -r /dev/mapper/Storage
$ sudo  xfs_db -c frag -r /dev/sde1

Here we’re asking xfs_db to open the file system in a readonly mode (-r) passing in a command (-c)  to get the file fragementation data (frag) for the device (/dev/*). When we use the frag command, it returns information only pertaining to the file data in the filesystem as opposed to the fragmentation of freespace (which we can guage with passing the freesp command). The output of the commands appear below for Zeus.

thushan@ZEUS:~$ sudo  xfs_db -c frag -r /dev/sde1
actual 189356, ideal 148090, fragmentation factor 21.79%

thushan@ZEUS:~$ sudo  xfs_db -c frag -r /dev/mapper/Storage
actual 406056, ideal 21584, fragmentation factor 94.68%

Wow! The LVM partition (which spans 4 drives) has around 95% fragementation! Yikes!!! The parition has quite a few Virtual Machine images, various large files (DV Captures etc). The “Backup” (sde1) on the other hand isnt as badly fragmented.

So right now we’ve found our problem and its time to fix it. First thing to do – and realise that we can fix this on a live running system – is to try and find a time where the partition will be used very little (like overnight) so you let its do its thing without unnecessary burden. Then lets make use of the File System Organiser utility (xfs_fsr) and ask it to reorganise our parition to the best of its ability.

$ sudo xfs_fsr -t 25200 /dev/mapper/Storage -v
$ sudo xfs_fsr -t 25200 /dev/sde1 -v

Now this is much simpler, the xfs_fsr utility is being told to reorganise /dev/* with a timeout (-t) of 7hrs  (60 * 60 * 7 = 25200) which is specified in seconds. Because I like to see how much is done I also specified the verbose output option (-v). Let it do its thing and hopefully when you return you will have the last bit of output showing the extents before, how many after and the inode, something like this:

extents before:5 after:1 DONE ino=4209066103
ino=4209066107
extents before:5 after:1 DONE ino=4209066107
ino=4209066101
extents before:4 after:1 DONE ino=4209066101
ino=4209066091
extents before:3 after:1 DONE ino=4209066091
ino=4209066093
extents before:3 after:1 DONE ino=4209066093
ino=4209066105
extents before:2 after:1 DONE ino=4209066105
ino=4209066143
extents before:27 after:1 DONE ino=4209066143

Now its time to go back and check how well the file system reorganising was:

$ sudo  xfs_db -c frag -r /dev/mapper/Storage

And the results?

thushan@ZEUS:~$ sudo  xfs_db -c frag -r /dev/mapper/Storage
actual 21652, ideal 21584, fragmentation factor 0.31%

Lovely! What a difference and you’ll notice the improvement immediately if you start moving or transfering files around.

Ideally, you may want to setup a cron task to let this process run (maybe with a lower timeout) overnight or when theres low-load. Whats great about the xfs_fsr utility is that its smart enough to remember where it finished up last time and continue from there. Its a shame Ubuntu doesnt do this already.

OpenSolaris 2008.11 has just been released, it encompasses some super cool new features and I’ve been waiting patiently to try this OS – need something new to learn!

The OpenSolaris 2008.11 operating system is a point of integration for the installation, desktop, and package management system projects on OpenSolaris.org. Today, the OpenSolaris 2008.11 live CD is available with the following feature updates:

ZFS Time Slider and Songbird;suspend/resume and CPU power management; Distribution Constructor and Prototype Automated Installer; WebStack with 64-bit MySQL, CherryPy, and DTrace for Ruby; GNOME 2.24, OpenOffice.org 3.0, and Firefox 3; Many F/OSS applications added, including top, sudo and Emacs; 700 additional man pages and Package Manager online help

Just a bit of background, OpenSolaris is based on Solaris, which was originally released by Sun Micro-Systems in 1991. Sun decided to release Open-Solaris to build a developer community around their Solaris product. Eventually it seems they will be basing technology for Solaris from OpenSolaris. So you know OpenSolaris will rock your world if its backed by Sun.

Download page for OpenSolaris 2008.11 or Direct Download of ISO and the 2008.11 Release notes.

Checkout the newly revamped OpenSolaris website, in particular the Learn area. Personally I’m looking forward to seeing the ZFS, Virtualisation Enhancements and DTrace loving.