KVM at Google
What Is KVM
KVM is the code in the Linux kernel that allows it to act as a hypervisor. A hypervisor creates and runs virtual machines. Being part of the kernel, KVM is a type 1 hypervisor, meaning it runs directly on hardware and as a consequence can take advantage of hardware capabilities to improve security and performance.
Why Google Cares
As a cloud provider, Google is a heavy user of virtual machines. Virtual machines allow the ability for many customers to share the same physical hardware without the possibility of interfering with each other and without the need to dedicate physical machines to each customers. Virtual machines also make it easy to present all customers with uniform tiers of service even though underlying hardware may vary.
Google pays a team of developers, including me, to make both internal and upstream upgrades to KVM. That means I get a Google salary to write open source software. Google loves when we can contribute our KVM upgrades upstream because it offloads the maintenance burden to the broader Linux community. Working with other developers from other companies saves labor costs over maintaining an internal fork of Linux and allows Google to build goodwill with the open source community.
The capabilities and performance of KVM directly affects Google's costs and competitiveness as a cloud provider.
What I Do
My high-level job description is to improve KVM's virtualization capabilities and performance.
Virtualization is a complex topic in computing requiring lots of specialized knowledge. The people I've met while working here are some of the smartest and most talented programmers I've ever met. The working memory alone required to mentally keep track of the hardware level, kernel level, and guest level software that is all acting in concert to make a virtual machine work is astounding. It's software complexity inception and it's insanely fun to hack on.
Here are some of the things that are required to fulfill my job duties
Hardware Function And Feature Awareness
Working with kernel code requires intimate knowledge of computer hardware, and KVM even more so. Virtual machines can be thought of as the ultimate exercise in information hiding. The guest OS should never know it is running in a virtual environment, so the hypervisor must comprehensively map all behaviors the guest expects from its virtual hardware onto actual hardware. That is no easy task, let alone doing it efficiently.
To accomplish this task efficiently, KVM must rely on virtualization help from the host CPU. That means understanding a lot about how CPU features work. I spend a lot of time referring to 5,000 page CPU reference manuals to make sure KVM is operating the features correctly. More CPU features are being added all the time, and customers want those features available to their virtual machines.
Linux Kernel Coding And Debugging
Linux is the largest and most successful open source project in history. It has millions of lines and a significant part of my job is understanding the division of labor inside the codebase and being very familiar with how they interact with virtual machines.
Of the millions of lines in the Linux kernel, I mostly focus on KVM for the arm64 architecture, which means the primary code I work with is about 70k lines. Even narrowing it down, the 70k lines of code is very complex. Kernel code forces an awareness of how much concurrency goes on in a modern OS. There are many kernel threads all accessing many different data structures, so mutexes abound. Not only that, kernel threads are subject to the scheduler, so you have to be aware of when your code mode be preempted at any time and the entire CPU state swapped out. Also, to squeeze every ounce of speed out, modern CPUs play fast and loose with instruction ordering and completion. For example, it's possible to write a register and that write is not actually visible to code after the write for a while unless you specifically force the CPU to wait for it with a barrier. Barriers are very inefficient and must be used sparingly. Kernel code is dangerous and frequently operates directly on hardware. If there is a mistake, the host machine will crash and must be restarted. That is a catastrophic result that is zealously protected against through stringent code review and huge batteries of tests.
This code is not easy to debug. You are often told as a userspace programmer that the compiler and the hardware are never at fault. As a kernel programmer that is not true. I encounter a hardware or compiler bug probably once every few months.
Userspace Awareness
I have to stare into the abyss of all the hardware and kernel code chaos listed above and squeeze all that complexity into behavior that is consistent and comprehensible to any userspace programmer that wants to make use of it. That means writing lots of documentation and doing lots of input validation. Not only this, but I and my coworkers have to look forward and figure out how new hardware capabilities can be used to better meet customer needs.
Kernel and hardware is the deepest possible level at which things can go wrong and the hardest to fix. If a customer ever knows my name, God help me.
Cross-Company Collaboration
Working in open source is a great opportunity to help the world with your work. I am lucky compared to so many Google employees whose work is born and dies inside the corporate silo.
On the other hand, to be accepted my commits must pass the scrutiny of much more experienced programmers that may be from other companies and have much different goals from me. If I am under pressure to ship fast, they don't care. They care about the code quality of the kernel. This leads to high quality code, but the hit to engineering velocity can be frustrating. Additionally you are talking to many people who are reviewing your code without ever having met you. The kernel mailing lists have a much deserved reputation for being mean sometimes.