In my last post I presented data that showed my virtualized Fedora 21.5 instance being faster than physical hardware running Ubuntu 12.04.
This was an interesting enough result to want to look further, but it’s not really a fair comparison. To be more equal, I need to at least run the workload on the same operating system version.
In this case, that means using two copies of Ubuntu 12.04 LTS, 64 bit.
Methodology:
Now, bear in mind that I’m not working in a corporate environment where I have easy access to multiple different, but equivalently specified platforms. I’m going to have to work with the systems I have, and that means VMs with occasional cross-checks with physical hardware.
In my experience, having precisely equivalent systems hasn’t been that important for my type of work: I’m looking for anomalous behavior, which generally results in large differences in performance. The signal-to-noise ratio is usually pretty high: We’re not normally looking at hard-to-detect ~3% performance variations with this type of problem.
Ultimately, I may end up doing 100% clean installs on both systems and benchmarking them side-by-side. I’ll probably know when I reach this point because I won’t be able to explain the differences any other way.
New data:
Here’s the results of doing 20 iterations of the Linux kernel compilation on this new operating system version:
The green histogram shows the new data. This new VM is substantially slower than any of the combinations I’ve tested so far.
In fact, if we consider median times, my Fedora 21.5 instance completes the kernel compilation job 750 seconds (12.5 minutes) faster than the Ubuntu 12.04 VM – that’s over 40% faster.
Equally, the VM of Ubuntu 12.04 is ~30% slower than Ubuntu 12.04 on physical hardware.
Note: I’m using the median time, not the average, because there may be an outlier in the new data off to the far right of the graph. Medians are not as skewed by outliers as averages are.
Note 2: I have not removed the suspected outlier, because it may equally be valid data. It was recorded under the same operating conditions as the other data, so it may possibly be something interesting – I’ll have to repeat the experiment and see if I get similar results.
Interpretation:
This sort of data is what makes performance analysis so interesting: I was initially surprised by my VM being faster than physical hardware. I’m now surprised by how slow my VM is with this version of Ubuntu.
It’s tempting to say “Well, VMs are just slower than real hardware, what did you expect?”, but that’s not quite accurate.
I could imagine that there is some overhead with using a hypervisor – especially a Type 2 hypervisor. However, I’ve never come across a clear analysis of how much performance really does degrade for different VMs on different platforms, so this counts more as common sense intuition than proven fact.
More importantly, the earlier Fedora 21.5 results show clearly that virtualized performance can be surprisingly good.
Next steps:
At this point, it seems that there’s something intrinsic about Ubuntu 12.04 that makes it run slowly on my VM when compared with physical hardware. I don’t know what it might be, so rather than theorizing, I need to find out.
It’s also possible that I got something slightly wrong in my 2nd hypervisor setup and that’s causing a performance problem – I’ll have to double check and run some more tests.
Assuming this result is reproducible, I’ll start to dive into what might be going on in my next set of posts. If not, it will be just as interesting to figure out what went wrong and why it has the impact it does!
nickjpavey 