Thushan Fernando Uncut

Linus Torvalds just released Linux Kernel 2.6.28 today as an Christmas present to all the good (and not evil) l33tle-big boys and gals around the globe. This brings a whole host of new cool bits and bobs thats sure to get people excited (Ext4 being one and the GEM Memory Manager) read all about it in the Linux Kernel Newbies guide.

Some addtional juicy reading incase your stuck with the inlaws for some ‘family’ time:

• Ext4
  • Linux Kernel Newbies guide to Ext4
  • Wikipedia
  • Phoronix Real World Benchmarks of Ext4
• Btrfs - Butter FS
  • Btrfs Information
  • Btrfs Architecture
  • Extensive and exhaustive examination of Btrfs Performance
• Indepth breakdown of 2.6.28

Enjoy!

Developer, Kernel / Internals, Linux/Unix, Operating Systems, hardware 2.6.28, btrfs, christmas, ext3, ext4, gem, gpu, Kernel, linus, linux, penguin, santa, satan, tux

It only feels like last month Sun released VirtualBox 2.0 and they’ve just released 2.1 which brings a plethora of additional goodies… from the changelog:

• Support for hardware virtualization (VT-x and AMD-V) on Mac OS X hosts
• Support for 64-bit guests on 32-bit host operating systems (experimental; see user manual, chapter 1.6, 64-bit guests, page 16)
• Added support for Intel Nehalem virtualization enhancements (EPT and VPID; see user manual, chapter 1.2, Software vs. hardware virtualization (VT-x and AMD-V), page 10))
• Experimental 3D acceleration via OpenGL (see user manual, chapter 4.8, Hardware 3D acceleration (OpenGL), page 66)
• Experimental LsiLogic and BusLogic SCSI controllers (see user manual, chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, page 70)
• Full VMDK/VHD support including snapshots (see user manual, chapter 5.2, Disk image files (VDI, VMDK, VHD), page 72)
• New NAT engine with significantly better performance, reliability and ICMP echo (ping) support (bugs #1046, #2438, #2223, #1247)
• New Host Interface Networking implementations for Windows and Linux hosts with easier setup (replaces TUN/TAP on Linux and manual bridging on Windows)

Some key things to note here, those “cool” people that run OS X can now get hardware virtualisation. Even if you have a 32bit host operating system your able to run 64bit hosts so long as you enable hardware acceleration on the CPU (AMD-V or Intel-VT) as VirtualBox’s Hypervisor requires this to work. A couple of other major additions - tested personally, include the enhanced virtualisation on the new Nahalem processors (Extended Page Table & Virtual Processor Identifier - see below) and the starting block for OpenGL (and later DirectX) Acceleration in XP and Vista. Testing this on OpenGL gave some decent performance though its still got a bit of work to do.

The move to include 3D acceleration is an interesting one, considering VMWare recently acquired Tungsten Graphics - who is the company behind Mesa, TTM memory manager and Gallium3D.  Interesting times ahead - as always

What’s an Extendable Page Table & that VPID thing???

Virtualisation in the Intel world comes in two flavours, the Intel VT-x and Intel VT-i Architectures. The VT-x is for IA-32 processors, whilst the VT-i is for Itanium processors.

Intel took a slice of the Virtualisation pie offered by AMD’s Pacifier architecture in implementing a method of translating ordinary IA-32 page tables from the guest-physical addresses to the host-physical addresses used to access memory. This way, guest’s can handle their own page tables directly and page-faults associated with them directly and minimize the (sizable) overhead associated with translating. This is known as Extended Page Tables (EPT).

Virtual Processor Identifiers (VPIDs) on the other hand allows a hypervisor (or a VMM) to assign a non-zero VPID to each virtual processor with the initial processor (VPID = 0) assigned to the hypervisor itself. This way, the CPU can use the VPIDs to tag translations in the Translation Lookaside Buffer (TLB) which removes the performance penalties associated with flushing TLBs on VM Entry and exit.

Both these two bits of technology (along with NMI-window exiting)  come on the Nahelem processor’s Virtualisation enhancments. If your interested in a more indepth explanation see the article Solving Virtualisation Challenges with VT-X and VT-I from the Intel Technology Journal.

Other Changes in 2.1

• VMM: significant performance improvements for VT-x (real mode execution)
• VMM: support for hardware breakpoints (VT-x and AMD-V only; bug #477)
• VMM: VGA performance improvements for VT-x and AMD-V
• VMM: Solaris and OpenSolaris guest performance improvements for AMD-V (Barcelona family CPUs only)
• VMM: fixed guru meditation while running the Dr. Web virus scanner (software virtualization only; bug #1439)
• VMM: deactivate VT-x and AMD-V when the host machine goes into suspend mode; reactivate when the host machine resumes (Windows, Mac OS X & Linux hosts; bug #1660)
• VMM: fixed guest hangs when restoring VT-x or AMD-V saved states/snapshots
• VMM: fixed guru meditation when executing a one byte debug instruction (VT-x only; bug #2617)
• VMM: fixed guru meditation for PAE guests on non-PAE hosts (VT-x)
• VMM: disallow mixing of software and hardware virtualization execution in general (bug #2404)
• VMM: fixed black screen when booting OS/2 1.x (AMD-V only)
• GUI: pause running VMs when the host machine goes into suspend mode (Windows & Mac OS X hosts)
• GUI: resume previously paused VMs when the host machine resumes after suspend (Windows & Mac OS X hosts)
• GUI: save the state of running or paused VMs when the host machine’s battery reaches critical level (Windows hosts)
• GUI: properly restore the position of the selector window when running on the compiz window manager
• GUI: properly restore the VM in seamless mode (2.0 regression)
• GUI: warn user about non optimal memory settings
• GUI: structure operating system list according to family and version for improved usability
• GUI: predefined settings for QNX guests
• IDE: improved ATAPI passthrough support
• Networking: added support for up to 8 Ethernet adapters per VM
• Networking: fixed issue where a VM could lose connectivity after a reboot
• iSCSI: allow snapshot/diff creation using local VDI file
• iSCSI: improved interoperability with iSCSI targets
• Graphics: fixed handling of a guest video memory which is not a power of two (bug #2724)
• VBoxManage: fixed bug which prevented setting up the serial port for direct device access.
• VBoxManage: added support for VMDK and VHD image creation
• VBoxManage: added support for image conversion (VDI/VMDK/VHD/RAW)
• Solaris hosts: added IPv6 support between host and guest when using host interface networking
• Mac OS X hosts: added ACPI host power status reporting
• API: redesigned storage model with better generalization
• API: allow attaching a hard disk to more than one VM at a time
• API: added methods to return network configuration information of the host system
• Shared Folders: performance and stability fixes for Windows guests (Microsoft Office Applications)

Performance & Updates

Overall, on the two different machines that I’ve tried the new 2.1 release on, they’ve both “felt” snappier (QX6850 and a Core i7 965E - architecture summary) but unlike the 1.6 release - which was somewhat flakey for me, 2.x releases of VirtualBox are solid.

Dont take my word for it, download and try it out.

Gets me a VirtualBox 2.1

Grab your copy and try it out.

• VirtualBox 2.1.0 for Windows hosts x86 | AMD64

• VirtualBox 2.1.0 for OS X hosts Intel Macs

• VirtualBox 2.1.0 for Linux hosts

• VirtualBox 2.1.0 for Solaris and OpenSolaris hosts x86 | AMD64

• VirtualBox 2.1.0 Software Developer Kit (SDK) All platforms

• VirtualBox 2.1.0 User Manual (updated 2008-12-17)

Give it a shot, heck try OpenSolaris 2008.11 on there just for kicks!

Developer, Kernel / Internals, Linux/Unix, OpenSolaris, Operating Systems, Tools / Products, Windows, hardware, software amd, amd64, hypervisor, ia32, ia64, Intel, linux, pacifier, PC, solaris, vanderpool, virtual, virtualbox, virtualisation, virtualization, virtualpc, vm, vmm, vmware, vt, Windows, x64, x86

Found an interesting run down of some information about the leaked builds of Windows 7 from WinHEC China by Robert McLaws. Theres also some juicy bits about the much loved boot-screen in the upcoming Windows 7 release.

Developer, Kernel / Internals, Operating Systems, Security, Windows, Windows 7, software boot, china, leaked, progress, vista, Windows, windows 7, winhec

Thats right, get it while its hawt! SP2 Beta for Windows Vista and Windows Server 2008 is available for your downloading pleasure. Weighing in at 338Mb its one download for both OS’s.

SP2 is an update to Windows Server 2008 and Windows Vista that addresses feedback from our customers and partners. By providing these fixes integrated into a single service pack, Microsoft provides a single high-quality update that minimizes deployment and testing complexity for customers.
In addition to all previously released updates, SP2 will contain changes focused on addressing reliability and performance issues, supporting new kinds of hardware, and adding support for several emerging standards. SP2 will also continue to make it easier for IT administrators to deploy and manage large installations of Windows Server 2008.

Service Pack 1 is a prerequisite for installing Service Pack 2. Please make sure that your system is running Service Pack 1 before you install Service Pack 2.
Windows Server 2008 SP2 Beta and Windows Vista Service Pack 2 Beta - Five Language Standalone version can be installed on systems with any of the following language versions: English, French, German, Japanese, or Spanish.

Technet areas:

• Windows Vista SP2 TechNet
  

• Windows Server 2008 SP2 TechNet

Download links:

Developer, Kernel / Internals, Operating Systems, Windows, software beta, microsoft, prerelease, server, service pack, sp2, vista, Windows, windows server 2008, windows vista sp2

OpenSolaris 2008.11 has just been released, it encompasses some super cool new features and I’ve been waiting patiently to try this OS - need something new to learn!

The OpenSolaris 2008.11 operating system is a point of integration for the installation, desktop, and package management system projects on OpenSolaris.org. Today, the OpenSolaris 2008.11 live CD is available with the following feature updates:

ZFS Time Slider and Songbird;suspend/resume and CPU power management; Distribution Constructor and Prototype Automated Installer; WebStack with 64-bit MySQL, CherryPy, and DTrace for Ruby; GNOME 2.24, OpenOffice.org 3.0, and Firefox 3; Many F/OSS applications added, including top, sudo and Emacs; 700 additional man pages and Package Manager online help

Just a bit of background, OpenSolaris is based on Solaris, which was originally released by Sun Micro-Systems in 1991. Sun decided to release Open-Solaris to build a developer community around their Solaris product. Eventually it seems they will be basing technology for Solaris from OpenSolaris. So you know OpenSolaris will rock your world if its backed by Sun.

Download page for OpenSolaris 2008.11 or Direct Download of ISO and the 2008.11 Release notes.

Checkout the newly revamped OpenSolaris website, in particular the Learn area. Personally I’m looking forward to seeing the ZFS, Virtualisation Enhancements and DTrace loving.

Developer, Its My Life, Kernel / Internals, OpenSolaris, Operating Systems, hardware, software OpenSolaris, operating system, solaris, storage, sun, zfs

Microsoft has just unleased the initial release of the Singularity 2.0 Research Development Kit (RDK). Singularity is a research operating system started around 2003 by Microsoft Research to write an OS in managed code. The inner-workings of Singularity taken from Wikipedia:

The lowest-level x86 interrupt dispatch code is written in assembly language and C. Once this code has done its job, it invokes the kernel, whose runtime and garbage collector are written in Sing# (an extension of C#) and runs in unsafe mode. The hardware abstraction layer is written in C++ and runs in safe mode. There is also some C code to handle debugging. The computer’s BIOS is invoked during the 16-bit real mode bootstrap stage; once in 32-bit mode, Singularity never invokes the BIOS again, but invokes device drivers written in Sing#, an extended version of Spec#, itself an extension of C#. During installation, Common Intermediate Language (CIL) opcodes are compiled into x86 opcodes using the Bartok compiler.

This new release brings some funky changes:

Its released under Microsoft’s shared source academic license which in basically means you can do what you like, just don’t make any money out of our hard work.

For convenience there’s even an ISO already baked ready to slap into a Virtual Machine

There are others that deviate from Singularity that tackle the use of a managed operating system slightly differently and I wrote about them a while ago.

.NET / CLR / C#, Developer, Kernel / Internals, Operating Systems, Windows .NET, acpi, amd64, c#, clr, cosmos, Java, Kernel, managed code, managed operating system, Operating Systems, shapos, singularity, smb, source

BD+ is the DRM system for Blu-ray discs, as Wikipedia puts it:

BD+ is a component of the Blu-ray Disc Digital Rights Management system. It was developed by Cryptography Research Inc. and is based on their Self-Protecting Digital Content concept. BD+ played an important role in the past format war of Blu-ray Disc and HD DVD. Several studios have cited Blu-ray Disc’s adoption of the BD+ anti-copying system as the reason they supported Blu-ray Disc over HD DVD.

One of the more humorous observations was that unlike DVD (which used DeCSS for its copy protection system) and AACS which powered the bulk of the HD-DVDs of the time that BD+ would uphold its protection for atleast the next 10 years. This may have been one of the key factors in the HD-Wars, but alas it seems someone  has found a way of traveling into the future and finding the break.

Oopho2ei (who claims is not a professional programmer :O) from the Doom9 forums along with a few others (bmnot, schluppo, Disabled, evdberg) have (it seems) successfully broken the BD+ protection scheme in a grand total of 5 weeks and 3 days (started on the 24th of August). They have restored the BD+ protected “The Day After Tomorrow”:

I am glad to announce the first successful restoration of the BD+ protected movie “The Day After Tomorrow” in linux. It was done using a blue ray drive with patched firmware (to get the volume id), DumpHD to decrypt the contents according to the AACS specification and the BDVM debugger from this thread to generate the conversion table. The conversion table is the key information to successfully repair all the broken parts in m2ts files to restore the original video content. This small tool was finally used to repair the main movie file “00001.m2ts” according to the conversion table.

To verify the correctness i compared my 00001.m2ts with the one AnyDVD-HD creates and they both match. The MD5 hash of this 30GB large file is in both cases “0fa2bc65c25d7087a198a61c693a0a72″.

Breaking the code is no simple feat, Oopho2ei and team has had to reimplement the VM that runs the BD+ protection layer and realises that there’s a fair chance that it could be blocked at a later stage and may phone-home:

There has to be some kind of firewall around the virtual machine which validates all communication between the ( potentially hostile ) content code and the outside world (traps and events). Part of the rules which are enforced by that firewall are the parameter checks on every trap call. It’s obvious that the traps and the event handling itself has to be carefully implemented. I believe this additional effort is necessary to prevent the content code from breaking out of it’s sandboxed environment and do nasty things like gathering user information and “calling home” when it detects an unlicensed emulator. So because these additional security measures make things more difficult i suggested to test this code first with the easy traps.

Even a guy from SlySoft (who makes the ever popular AnyDVD-HD product) chimes in early on but backs off after realising he could well get the sacker.

I’ll just say: due to certain properties of BD+, once you’re past a certain point, you can handle it pretty much without reversing - BD+ itself then helps you out - on any player

…

Actually you’d have to know how BD+ really works, to know what I meant (and even then you probably wouldn’t ).
But if I start unraveling that, I’d be finding myself looking for a new job by next week

Love this bit in one of Oopho2ei posts:

I would like to stress again that this project wasn’t intended to circumvent copy protection and promote piracy. This can already be done using commercial software like AnyDVD-HD. Instead this project was an attempt to enable users of open source operating systems (like linux) to playback their BD+ protected discs without having to use proprietary software. Furthermore only two movies “I Robot” and “The Day After Tomorrow” have been proven to be handled correctly so far. Obviously there is still a lot of debugging to be done.

Classy! Download a copy of the BDVmDbg build for educational reasons and try PortableBDVM which comes in C99 source form.

Developer, Kernel / Internals, Linux/Unix, Operating Systems, Security, Windows, hardware, software aacs, anydvd, bd, bdvmdbg, blueray, bluray, crack, dcma, decss, Disney, doom9, drm, dvd, h.264, hack, hacked, hd, hdcp, hddvd, Intel, Matsushita, microsoft, Oopho2ei, Panasonic, portablebdvm, reverse, slysoft, sony, video, virtual machine, vm

If you like discussions about deep internals you’ll most definately have subscribed to the Going Deep series on Channel 9. Today they just released a fascinating interview with Kernel Guru, Mark Russinovich - of Sysinternals fame, who is now a Technical Fellow at Microsoft. One of my favourite books would have to be Windows Internals 4th Edition, and reference it quite frequently. Cant wait for the 5th edition!!!

One very important change in Windows 7 kernel is the dismantling of the Spin Lock Dispatcher and redesign and implementation of its functionality into separate components. This work was done by Arun Kishan (you’ve met him here on C9 last year). The direct result of this great work is that Windows 7 can scale to 256 processors and enabled the great Landy Wang to tune Windows Memory manager to be even more efficient than it already is. Mark also explains (again) what MinWin really is (heck, even I was confused. Not anymore…). MinWin is present in Windows 7.

There are some really interesting topics covered in this video, especially the content behind the scheduler and the thread dispatcher.

Channel 9 Going Deep: Inside Windows 7

Download Offline versions: WMV | WMV HD | MP4 (iPod) | ZUNE

.NET / CLR / C#, Developer, Kernel / Internals, Operating Systems, Security, Windows, Windows 7, software contention, cores, dispatcher lock, internals, Kernel, lock, mark russinovich, minwin, multicore, multiprocesser, scheduler, sleep, spin lock, spinning, thread, threading, wake, win7, Windows, windows 7