Storage operations by the Window Manager

I’m partial to minimal window managers (WMs) like FluxBox and IceWM.

When one is working in a corporate environment on shared infrastructure, I think there are advantages to using a more stripped down WM:

• Interactive machines often host many users (an interactive session typically doesn’t take many machine resources), so less load benefits everyone.
• The stripped down WMs don’t use transparency or subtle shading. As a result, they tend to be more suitable for remote access tools like VNC or NoMachine’s NX, where WAN latency and bandwidth makes a difference to usability.
• I have seen shared machines where multiple Gnome desktops sessions synchronized their background tasks, leading to load spikes.

The synchronized gnome problem was interesting. It tended to show up as users complaining that their interactive sessions froze up every few minutes.

We did some debug, and we discovered that Gnome sessions appeared to be traversing the files containing language options fairly frequently. There are quite a few files for localization, and it seems that all the sessions did it at the same time. This caused a spike in disk IO, which showed up as reduced interactive performance.

Now, this happened a while back on RHEL5.3 (I think?) machines with a large number of users on each box.

Based on my experience a few years back with an old OS version, I can’t generalize and say that all Gnome WMs have this problem. I also don’t know what other modern WMs get up to in the background either.

This blog is about analyzing problems rather than making assertions. So, instead of speculating, let’s find out!

This is a good chance to get into some of the more advanced debugging tools, which is what this blog promised to talk about.

The next post will show you how to get RedHat’s SystemTap tool working on Ubuntu 14.04 LTS.

Manually installing the VirtualBox “GuestAdditions” package

In the previous post I described how to locate and install the Oracle VirtualBox GuestAdditions package.

In this post, I’m going to walk through my experience of doing it manually.

The system I’m using for this is:

• VirtualBox version 4.3.20
• Ubuntu 14.04 LTS 32 bit guest OS
  • VirtualBox doesn’t want to run 64 bit guest OSes without some BIOS tweaks
• Mac OS X (Yosemite) host system

The Ubuntu guest OS is a completely clean install.

Using a clean install is actually fairly representative of real life. You typically only do this job once when you first set up the OS, so you can get a properly sized screen.

Step 1: Installing ‘dkms’:

The first thing to do is get the DKMS framework installed. You can read more about the system here:

• http://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
• https://help.ubuntu.com/community/DKMS

This is pretty straight forwards and works perfectly on my clean 14.04 LTS install:


sudo apt-get update
sudo apt-get install dkms


If you get into trouble, it’s likely because your Linux install doesn’t have the source code headers for your kernel version available. I’ll be covering this as part of future  posts on how to build some of the more advanced debug tools.

Step 2: Mounting the Guest Additions ISO image

You’ll need to mount the “VBoxGuestAdditions.iso” image described in the previous post into your running VM. In this case, you should mount the ISO while the guest OS is running.

I found that the VirtualBox ISO mounting dialog doesn’t allow you to right-click into the VirtualBox application, so I couldn’t use the ISO in its pre-packaged location.

Given that I know where the ISO lives in the VirtualBox directory, I copied the ISO to an ordinary directory in my home area. The ISO selection dialog can see it in its new location, and will mount it correctly.

Interestingly, if you mount the CD before starting the VM, the boot loader doesn’t try to boot off the ISO. At least, if it does, it handles the error cleanly and doesn’t throw a user-visible error, which is nice.

The Ubuntu 14.04 LTS version I’m using wants to auto-run the ISO and will present you with a pop-up box asking whether to go ahead. Since this is a post about manually installing the module, I’ll decline this.

Step 3 : Doing the installation

For this combination of hypervisor and guest OS, the ISO will be mounted in this location inside the guest OS:


/media/${USER}/VBOXADDITIONS_4.3.20_96996


Take a note of this path, because you’re going to need to ‘cd’ to it in a root shell in the next step.

You can initiate the installation as follows:


sudo bash (to get yourself a root session)
cd <path to the GuestAdditions ISO>
sh ./VBoxLinuxAdditions.run
reboot


This went perfectly for me.

Once the guest OS comes back, you should be able to resize your virtual window (or make it full-screen) and have Ubuntu give you the right display size.

Finding and installing guest additions for Oracle VirtualBox

I’m going to be using a couple of platforms for this blog:

• Virtualized instance of Ubuntu 14.04 LTS
  • Running under Oracle VirtualBox 4.3.20, with a Mac OS X host (Yosemite)
  • This configuration requires that the guest OS must be 32 bit.
• Physical instance of Ubuntu 14.04 LTS
  • I don’t have the hardware for this yet, but it’s likely to be an AMD A10 based system.

Having physical hardware will be important for this blog, because we’ll be needing access to the CPUs performance counters and debug hardware.

Still, you can get a long way with virtualized systems (not all advanced tools need hardware access), so I’ll get started with a VM.

(By the way, kudos to Oracle for providing a perfectly good hypervisor for free. It works on multiple platforms and has enough features to allow you to get work done.)

The “GuestAdditions” package:

I’ve installed VirtualBox many times, and every time I do it, I get hung up on getting a decent screen resolution. The key is to get the “GuestAdditions” package installed.

Now, the problem is that it’s actually surprisingly hard to find the package. There is a VirtualBox menu item that should allow you to automatically install it into your guest OS, but I’ve had varying results with that method.

It turns out that in VirtualBox 4.3.20, the GuestAdditions are packaged with the application. This is really good to know, because finding Oracle’s download site is a bit tricky.

Locating the included package:

OS X applications are not simply a binary executable file. They contain a bunch of resources that the program needs to run – a bit like the standard ‘bin’, ‘lib’ and ‘share’ directories you’ll see with packaged linux executables.

Navigate to the VirtualBox application with your finder, you can open the contents with a right click, then select “Show Package Contents”. This will drop you into a folder where you can see all the working parts of the application.

Navigate to “<app location>/Contents/MacOS” and you’ll find the file “VBoxGuestAdditions.iso”. This is what you’ll need.

Note : This post is specific to Mac OS X. I don’t know what you’ll find on other platforms, so you may need to take the manual approach described below.

Installing the package:

It’s worth giving the standard GUI based approach a try first.

Start your VM, then navigate to the top menu bar and select “Devices | Insert Guest Additions CD image”. This ought to do the work automatically. Reboot the VM to make sure the changes have been fully applied.

This approach just worked fine for me, but that hasn’t always been the case.

Fetching the package manually:

If you end up needing to fetch and install the GuestAdditions package manually, here’s what to do.

First of all, check to see if your version of VirtualBox already contains the GuestAdditions ISO, as described above. If it does, then you can use that.

If not, you can download the package from here:

http://download.virtualbox.org/virtualbox/

You’ll need to select the directory that matches the version of the VirtualBox hypervisor you’re using.

In my case, I’m using version 4.3.20, so I’d select the package from here:

  VBoxGuestAdditions_4.3.20.iso

(Note: WordPress doesn’t seem to allow me to get the text path in the link here.)

Building the package manually:

Installing the GuestAdditions seems to require building a new kernel module. This means that getting the package to install is a two step process.

First you need to get your system ready to build kernel modules, as described in Oracle’s manual here:

https://www.virtualbox.org/manual/ch02.html#externalkernelmodules

Then you’ll need to build the additions themselves, as described in section 4.2.2.1 of Oracle’s VirtualBox manual,  here:

https://www.virtualbox.org/manual/ch04.html#idp54932560

The next entry will describe actually doing this, and any pitfalls I encounter – stay tuned!

What makes performance analysis hard?

Performance analysis of complex, running systems is famously hard. Why?

Firstly, you probably don’t know about any latent scaling problems – if you did, then you’d already have fixed them before the system started to misbehave.

Instead, performance problems tend to be subtle at first, maybe insignificant enough to discount as a glitch or just normal behaviour. Then, performance falls off a cliff and you start getting the dreaded 3am pager calls.

The software installed on a production system often does not include all the analytical tools you might want. Even if they were installed, some advanced debug tools can be pretty invasive, so you may not want to launch them, for fear of doing more harm than good.

If you’re running a serious professional installation, especially if you have operations worldwide, then there’s no good time to do in-depth analysis. It’s always the workday for some (possibly quite large) group of users, so you can’t take the system down and run some experiments. Alternatively, the system could be part of a large automated workflow, and taking it down could stall or destabilize the rest of the pipeline.

Finally, you may not actually understand the code that’s running on the machine. It could be a commercial application where you can’t get the source, or it could be a large open source system where it’s not practical (or relevant to your business) to fully understand the code base. It could be a well-understood system that suddenly starts showing unexpected new behavior.

This leaves you with a problem: How can you analyze what’s going on in a complex system with a fairly ordinary set of diagnostic tools? How do you do it reasonably quickly? What kinds of solutions work well for different types of problem?

That’s what this blog is about. I hope you find it useful.

Welcome to Nick Pavey’s Blog

This blog focuses on how to identify, analyze and hopefully, fix performance problems on production Linux systems.

I will focus more on server workloads, but the information in here will be useful if you’re encountering problems on your desktop too. Also, some of the higher level information will be useful on other platforms as well.

I’m going to be adding information over time, so watch this space! In the meantime, let me leave you with one of my favorite sayings about debugging:

“Debugging is anticipated with distaste, performed with reluctance, and bragged about forever!”