Thushan Fernando Uncut

Now that I’ve decided what I want out of the server (and the hardware I’ve got), its time to workout what operating system to run the system on. Currently, ZEUS is running on Ubuntu Gutsy (7.10) which is running LVM with an XFS volume holding approximately 2.5Tb worth of data. There’s a cron job that defrags the XFS volume to keep things in order.

The Operating System

As the operating system is no longer maintained (my oversight into how long it would survive) I have to find an OS that supports the hardware platform without hacky hacky bits (and by this I mean avoiding buggy ACPI and issues with the NForce4 chipset and IRQ problems) and has a file system that will benefit long term.

There were a few considerations:

• Ubuntu 8.04.x LTS
  I like Ubuntu, I’m comfortable with the user land and find the Debian package system (in particular the dependency resolving) most impressive. Hardware is well supported and 8.04.3 (at the time of writing) boots on the hardware I originally selected (Intel) and the new configuration I recently selected (AMD). I could most definitely use Ext4 but the problems with data-loss (which I’ve reproduced on several occasions on desktop machines) scare me.FileSystem: I’d have to adopt either XFS or Ext4 on an LVM to factor in future-proofing, maybe get some fakeRAID happening for redundancy.
  Installation: comes with a Server edition that’s bare bones allowing it to be a minimalistic installation which is always nice!
• Ubuntu 9.04
  Initially when I started to rebuild Zeus back in April I wanted to use Ubuntu 9.04, I was really excited about Ext4 and the promise of a brand-spanking new file-system and what it would bring to the table. Unfortunately after using Ext4 with 9.04 I’ve come to realise its probably not the wisest to trust your data with it just yet – unless you get yourself a UPS! Laptop seems to be chugging nicely though.Installation: Like LTS, comes with a Server edition that’s bare bones allowing it to be a minimalistic installation which is always nice! (copy/paste!) Unfortunately picking 9.04 when 9.10 is just around the corner is not going to be ideal, I’ll be stuck with where I am right now in a year or so.

• CentOS 5.3 or wait for CentOS 5.4 coming soon!
  CentOS is based off Redhat Enterprise Linux and having only started using CentOS after joining HSD (though my first splash in Linux was with Bowtie Redhat Linux 6.0 – I still have the CD!) I find the RHEL clone to be stable and solid as a rock! CentOS 5.4 will (thanks to RHEL 5.4) bring some KVM goodness that is a high priority on my list.

  FileSystem: Ext4 is not in the RHEL5 stream (have to wait for RHEL6 – which is a good thing!) so the best bet here is to use XFS with LVM.
  Installation: Customised via kickstarts or setup a cobbler server to install via PXE.

• OpenSolaris
  OpenSolaris has some really exciting technologies that are worth a look at, but in particular is ZFS. ZFS breaks all the rules – and for a good reason. I’ve yet to really use Solaris – quite cosy with Linux, but there’s nothing like getting used to another OS by jumping straight into the deep end!

  FileSystem: ZFS all the way baby!
  Installation: Installation is via a LiveCD, I’ve looked into customising the installation so it removes some bits I don’t which I’ll discuss later but its quite fiddly.

So in case the sudden influx of OpenSolaris posts didnt give you the hint, I decided on OpenSolaris to power the new iZeus 2.0, actually no that sounds lame, zeusy will be the new ZEUS until ZEUS is retired in which case zeusy becomes zeus (confused?).

Why ZFS?

ZFS is one of those file-systems you look at and think, wow! Why didn’t anyone else think of that before?

• Very simple administration – you only use two commands, zpool and zfs.
• Highly scalable – 128-bit means we can hold 16 exabytes or 18 Million terabytes worth of data! More porn for you! XFS can no doubt handle the TBs we use for our home boxes now, but no-chance you can get the performance or benefits of ZFS in Ext3/Ext4 or XFS.
• Data integrity to heal a filesystem (no fsck’ing around!) – 256bit checksuming to protect data, if ZFS detects a problem it will attempt to reconstruct the bad block and continue on its merry way (utilising available redundancy)
• Compression – you can elect to compress a particular file-system or a hierarchy just by setting one command! I’m thinking things like logs here.
• No hardware dependency – JBOD on a controller, let ZFS maintain the RAID volumes in software. Checkout Michael Pryc’s crazy adventure with ZFS using USB thumb drives and Constantin’s original voyage with USB drives! RAID-Z is essentially RAID-5 without the write-hole problems has plagued it if power is lost during a write, it can also survive a loss of a drive (with RAIDZ-2 you can loose two drives).
• Happy snaps for free! Snapshot (a live) file-system as many times as you like, again one easy command. Its like that tendency to hit {CTRL+S} when your working in Windows from back in the days of Windows 9x, snapshot regularly!

So ZFS sounds much like marketing spiel right now, best thing since sliced bread, cooler than a cucumber, and you’d be right it is cool and the best thing since filesystems came to being. Over the coming days I’ll post some more on my musings with ZFS – keeping in mind that I’m still learning these things. It helps to have lots of hardware to play with, but even if you don’t, you can knock up a virtual version of OpenSolaris in VirtualBox, create some virtual disks and try it out.

There are a few caveats that I’ve come across though using ZFS, one is memory! ZFS will try and cache as much data as it can in RAM, so if you have 8Gb of RAM (as I have in this box) it will happily use as much of it as it can afford. Rightfully so, I was getting ~96MB/s transfering a 16Gb MPEG from one box to the other over our Gig link (thats from one end of the house to the other!) mind you this was just a test configuration using 2x 74Gb Western Digital Raptors (WD740ADFD) in a RAID-0 style hitting a single 150Gb Western Digital Raptor (WD1500ADFD). They could have gone much higher, but I was happy with that.

There are also (as of writing) no recovery tools for ZFS, but these are slated to arrive soon (Q4 2009) which is quite scary after you read this post about a guy loosing 10Tb worth of data, however a possible revert to an older uberblock may fix some problems.

Virtualisation

Initially I wanted to concentrate quite a bit on Virtualisation, I tried Xen on OpenSolaris. It was quite easy to setup a Xen Dom0 in OpenSolaris but with the 2009.06 release you had to tweak the Xen setup a bit. I wasn’t too enthusiastic about using Xen after seeing the performance lag in Windows in my musings. Instead I’m opting for my crush, VirtualBox.

So why use VirtualBox when you can get a bare-metal hypervisor? Firstly, performance seems to be sluggish with Xen for me (I didn’t investigate this too much), secondly I want to be able to run the latest and greatest OS’s out without worrying about upgrading Xen (I’m a sucker for OS’s!). VirtualBox development has accelerated at a feverish pace, I started with VirtualBox 1.3 in 2007 and its come an insanely long way since then. When a new release comes along, its as easy as updating VirtualBox and getting all the benefits. Plus with SunOracle‘s backing of VirtualBox you know things are going to work well on OpenSolaris, the Extras repository of VirtualBox makes it as easy as doing a pkg update.

I’m still quite intrigued by the way KVM is heading and how it will pan out, but for the future zeus, it will be VirtualBox.

I’ve been looking at upgrading our existing home server from the archaic (and unsupported!) Ubuntu Gutsy (because I was feeling gutsy at the time) to something newer, fresher and that will last me atleast another 2 years. This is purely for my documentation.

Current Setup

Currently running an AMD setup with Ubuntu Gutsy (7.10) – I didn’t think it would last this long, honest! Ubuntu 6.06 had too many issues with the hardware/driver incompatibilities.

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=7.10
DISTRIB_CODENAME=gutsy
DISTRIB_DESCRIPTION="Ubuntu 7.10"

On an ASUS A8N-SLI Deluxe motherboard (because you know, servers need SLI!) sporting a AMD Athlon64 3200+ (the only AMD CPU at home!) with 2Gb of RAM (hey, DDR1 wasn’t cheap enough!)

lspci

00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev f3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev f2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev f3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev f3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7300 GT] (rev a1)
05:06.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
05:07.0 RAID bus controller: Silicon Image, Inc. Adaptec AAR-1210SA SATA HostRAID Controller (rev 02)
05:0a.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
05:0b.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
05:0c.0 Ethernet controller: Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller (rev 13)

/proc/cpuinfo

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 47
model name      : AMD Athlon(tm) 64 Processor 3200+
stepping        : 2
cpu MHz         : 1000.000
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm ts fid vid ttp tm stc
bogomips        : 2011.59
clflush size    : 64

This faithful boxen has been the primary source of our fileserver (XFS+LVM 3Tb) – used internal to our house and also by others who upload their stuff to be backed up. Subversion repositories, Apache/LightHttpd test servers for PHP work, Virtualisation for Windows 2003, 2000 and SqlServers running for testing and several other things (think: TeamCity, Continous Integration tools, Confluence etc). Its also been damn convenient when your at work or on holidays to be able to login, muse about via SSH and even fix things remotely.

Needs & Wants

The new server will need to fufil the following roles:

• Function as a NAS to continue to offer backup (via users home directories) and storage options
  • No file-system constraints asside from no Ext3 or ReiserFS.
• Offer the ability to still run Virtual Machines, need to virtualise CentOS, Ubuntu and Windows for testing, they’ll be running in  Bridged mode
• No real need for a Gui (I can consider myself a little l33t than a few years ago)
• Run a Subversion repository (not that hard!)

The idea is to have a bare bones operating system install and have the virtual machines handle the hard and ugly work – webservers to test things, servers to try development deployments (java) and other bits and pieces. The core OS just has to manage the NAS and allow the ability to SSH in to offer subversion access.

Hardware

The hardware I’ve picked from things I had around the place, the only thing I’ve bought is just new sticks of RAM.

• Motherboard: ASUS P5QL-PRO
  This board offered some excellent specifications via the P43 chipset, the things I looked for was the number of SATA ports ‘out of the box’ – 6 native SATA2, the number of 1x PCIe slots (2!) for future addions of PCIe SATA adapters and the maximum amount of memory possible (8Gb). Oh ofcourse, something cheapy and that can run the CPU I had around. A Gigabit NIC was also important (dual would be better!) but if it wasn’t supported I had a trusty Intel PRO 1000MT Server PCI cards to fill the void – almost everything supports them (e1000)!
• CPU: Intel Core-2 E6750 – 2.66Ghz (65W TDP, VT)
  Importance was Intel-VT support, low TDP and a dualcore thats not too high.
• RAM: Corsair TWIN2X4096-6400C5 (4Gb kit x 2 = 8Gb)
  Cheapy cheapy, twice the fun of a regular kit, slightly higher CAS, but who CAreS this isnt being overclocked.
• Graphics: ASUS 9400GT PCI-Express
  The cheapest graphics card to be found at the legendary& award winning computer store MSY Technologies. Depending on how the drivers go (I’m usually biased towards ATI for all Linuxes) I might endup paying for a ATi card later.

Next up the investigation, be warned though I started this initially back in June/July (possibly a bit earlier).

So today we had to move our clocks forward 1 hour (at 2am). Just realised that our Ubuntu box (zeus) was not displaying the right time to begin with (prior to the change). So to keep your computer clock in time with the world, use ntp.

$ sudo ntpdate [timeserver]

I use the Melbourne AAPT time-server but you can find another here or if your an aussie, use these:

Configure your /etc/ntpd.conf with

driftfile /var/lib/ntp/ntp.drift
server ntp.mel.connect.com.au

So how far off was zeus?

4 Oct 21:39:46 ntpdate[24844]: step time server 192.189.54.17 offset -1225.284656 sec

eeek!

\

Source: http://xkcd.com/644/

If you haven’t heard already, Redhat has released the eagerly anticipated 5.4 release of Redhat Enterprise Linux at their Redhat Summit in Chicago. As expected, Redhat looks to have moved from using Xen as their favoured virtualisation hypervisor to using KVM (which is an integral part of the Linux Kernel). All this will eventually go into RHEV.

All the changes in this release are documented in the  Release Notes, unfortunately Ext4 is still not considered usable in this release (they’re targetting for RHEL6 possibly).

So what of the RHEL clone CentOS? Possibly a 2-4 week delay it seems. WOO! In the meantime, upgrading from 5.3 is easy peasy.

I’ve been working heavily with Red Hat Enterprise Linux (and subsequently CentOS) the past few months (shh! dont tell my MSFT homey!) and one of the great things about CentOS and RHEL is that they both install using LVM – which is a helluvah lot easier when time passes and you realise your running out of space on a drive.

But today I had to recover some data from an LVM partition and copy over some bits to another partition without actually booting the CentOS install (it was bj0rked by yours truely!). What to do? Throw in a Ubuntu LiveCD (or another) and just mount the partitions

First thing we need to do is install LVM – remember we need to be sudo for these to work.

$ aptitude install lvm2

Then scan for any available physical volumes on any of the drives.

$ pvscan

Scan for any Volume Groups that may be present.

$ vgscan

Now activate any of the Volume Groups that it finds, running this makes the logical volumes known to the kernel.

$ vgchange –available y

Then let it scan for any Logical Volumes on any drives

$ lvscan

After running the logical volume scan it will show the path to the LVM mount path, for my boxen it gives something like this

ACTIVE            ‘/dev/LVM/Data‘ [5.26 TB] inherit

You simply mount the path specified and browse like normally

$ mount /dev/LVM/Data /mnt

Enjoy.

Thats right, that wasn’t a typo. Some crazy boffins at Sandia National Laboratories in Livermore, have run more than a million Linux kernels as virtual machines out of which 20,000 can be run simultaneously! Why on earth would they attempt such feats?

Prehaps this XKCD may jog your memory

Yep, just about:

The technique will allow them to effectively observe behaviour found in malicious botnets, or networks of infected machines that can operate on the scale of a million nodes.

Insane!

Valerie Aurora (such a cool name!) takes a look into the history of Btrfs, well written and easy to follow.

Sun is prepping up the release of VirtualBox 3.0 which should be out soonishly (before the end of the month if everything is on track!). Amongst the top new bits of functionality is the guest multi-processing (SMP) support, you can now offer your virtual machines upto 32 virtual CPUs, which at last count was the highest by any hypervisor (VMWare can offer a maximum of 2). Not only that, but Windows guests get Direct3D 9 acceleration, bi-directional OVF support (still need to sysprep drivers) and OpenGL 2.0 support for Linux, Solaris & Windows guests.

VirtualBox 3.0 Beta 2 was released yesterday, from the Beta 2 forum post:

Changes since Beta 1: (regressions and fixes for new features only; additional fixes below)

• High CPU usage with certain idle Windows guests (e.g. XP SP2)
• Performance improvements for SMP guests
• Reset issues with SMP guests
• Windows 7 x64 boot problems with SMP
• Windows 7 shutdown hangs with SMP
• Hanging VM when unmounting an ISO with SMP guests
• Several OpenGL & Direct3D related fixes

Version 3.0 will be a major update. The following major new features were added:

• Guest SMP with up to 32 virtual CPUs (VT-x and AMD-V only)
• Windows guests: ability to use Direct3D 8/9 applications / games (experimental)
• Support for OpenGL 2.0 for Windows, Linux and Solaris guests

In addition, the following items were ?xed and/or added:

• Solaris hosts: allow suspend/resume on the host when a VM is running (bug #3826)
• Solaris hosts: tighten the restriction for contiguous physical memory under certain conditions
• Mac OS X hosts: fixed guest PAE
• Linux hosts: kernel module compile fixes for 2.6.31 (bug #4264)
• VMM: ?xed occassional guru meditation when loading a saved state (VT-x only)
• VMM: eliminated IO-APIC overhead with 32 bits guests (VT-x only, some Intel CPUs don’t support this feature (most do); bug #638)
• VMM: ?xed 64 bits CentOS guest hangs during early boot (AMD-V only; bug #3927)
• VMM: performance improvements for certain PAE guests (e.g. Linux 2.6.29+ kernels)
• VMM: some Windows guests detected a completely wrong CPU frequency (bug #2227)
• VMM: fixed hanging and unkillable VM processes (bug #4040)
• GUI: added mini toolbar for fullscreen and seamless mode (Thanks to Huihong Luo)
• GUI: redesigned settings dialogs
• GUI: allow to create/remove one host-only network adapters
• GUI: allow to create/remove more than one host-only network adapters (non Windows hosts)
• GUI: display estimated time for long running operations (e.g. OVF import/export)
• GUI: Fixed rare hangs when open the OVF import/export wizards (bug #4157)
• 3D support: fixed VM crashes for client apps using incorrect opengl states
• VRDP: support Windows 7 RDP client
• Networking: ?xed another problem with TX checksum of?oading with Linux kernels up to version 2.6.18
• NAT: allow to con?gure socket and internal parameters
• VHD: properly write empty sectors when cloning of VHD images (bug #4080)
• VHD: ?xed crash when discarding snapshots of a VHD image
• VHD: fixed access beyond the block bitmap which could lead to arbitrary crashes
• VBoxManage: ?xed incorrect partition table processing when creating VMDK ?les giving raw partition access (bug #3510)
• OVF: several OVF 1.0 compatibility ?xes
• Virtual mouse device: eliminated micro-movements of the virtual mouse which were confusing some applications (bug #3782)
• Shared Folders: sometimes a ?le was created using the wrong permissions (2.2.0 regression; bug #3785)
• Shared Folders: allow to change ?le attributes from Linux guests and use the correct ?le mode when creating ?les
• Shared Folders: some content was incorrectly written under certain conditions (bug #1187)
• Shared Folders: ?xed incorrect ?le timestamps, when using Windows guest on a Linux host (bug #3404)
• X11 guests: a number of shared clipboard fixes
• Linux guests: Guest Additions support for SUSE Linux Enterprise Desktop 11
• Linux guests: new daemon vboxadd-service to handle time syncronization and guest property lookup
• Linux guests: implemented guest properties (OS info, logged in users, basic network information)
• Windows host installer: VirtualBox Python API can now be installed automatically (requires Python and Win32 Extensions installed)
• USB: Support for high-speed isochronous endpoints has been added. In addition, read-ahead buffering is performed for input endpoints (currently Linux hosts only). This should allow additional devices to work, notably webcams
• Web service: fixed inability to handle NULL pointers for object arguments, which are valid values for a lot of APIs, in both the raw and the object-oriented web service.
• Web service: object-oriented bindings for JAX-WS did not exhibit interface inheritance correctly, fixed
• Web service: added support for IDisplay and IGuest interfaces, which were previously unavailable
• Registration dialog uses Sun Online accounts now

As a VBox junkie I’ve moved my lappy to use the 3.0 Beta 1 and Beta 2 releases and Beta 2 1 day into musing seems quite stable.

VirtualBox-3.0.0_BETA2-49051-Linux_amd64.run                         46M
VirtualBox-3.0.0_BETA2-49051-Linux_x86.run                           46M
VirtualBox-3.0.0_BETA2-49051-OSX.dmg                                 61M 
VirtualBox-3.0.0_BETA2-49051-Win.exe                                 67M

Try it out, but be warned its not yet ready for production.

Linus has released 2.6.30 of the kernel, list of changes are available in the Linux Kernel Newbies guide.

This version adds the log-structured NILFS2 filesystem, a filesystem for object-based storage devices, a caching layer for local caching of NFS data, the RDS protocol which delivers high-performance reliable connections between the servers of a cluster, a distributed networking filesystem (POHMELFS), automatic flushing of files on renames/truncates in ext3, ext4 and btrfs, preliminary support for the 802.11w drafts, support for the Microblaze architecture, the Tomoyo security module, DRM support for the Radeon R6xx/R7xx graphic cards, asynchronous scanning of devices and partitions for faster bootup, MD support for switching between raid5/6 modes, the preadv/pwritev syscalls, several new drivers and many other small improvements.

One interesting change (amongst the many) is that we have this new feature called Fastboot. Essentially, when we boot right now, there is significant cycles wasted waiting for the device probing to complete. From Johnathan Corbet’s article on LWN:

There are many aspects to the job of making a system boot quickly. Some of the lowest-hanging fruit can be found in the area of device probing. Figuring out what hardware exists on the system tends to be a slow task at best; if it involves physical actions (such as spinning up a disk) it gets even worse. Kernel developers have long understood that they could gain a lot of time if this device probing could, at least, be done in a parallel manner: while the kernel is waiting for one device to respond, it can be talking to another. Attempts at parallelizing this work over the years have foundered, though. Problems with device ordering, concurrent access, and more have adversely affected system stability, with the inevitable result that the parallel code is taken back out. So early system initialization remains almost entirely sequential.

This new release attempts to address this problem.

Arjan hopes to succeed where others have failed by (1) taking a carefully-controlled approach to parallelization which doesn’t try to parallelize everything at once, and (2) an API which attempts to hide the effects of parallelization (other than improved speed) from the rest of the system. For (1), Arjan has limited himself to making parts of the SCSI and libata subsystems asynchronous, without addressing much of the rest of the system. The API work ensures that device registration happens in the same order is it would in a strictly sequential system. That eliminates the irritating problems which result when one’s hardware changes names from one boot to the next.

How well it does it, I guess we’ll have to wait and see. But here’s a bit of a tidbit in the kernel for the new Microblaze implementation.

void __init setup_cpuinfo(void)
{
struct device_node *cpu = NULL;

cpu = (struct device_node *) of_find_node_by_type(NULL, "cpu");
if (!cpu)
printk(KERN_ERR "You don't have cpu!!!\n");

printk(KERN_INFO "%s: initialising\n", __func__);

DUDE, You dont’ have cpu!!!