**coltonlewis.name: Linux Kernel Development Advice [Org] All L1 (Kernel Hacker Mode) ---

Linux Kernel Development Advice

A friend of mine wanted to get into sending patches to the Linux Kernel and these are the ten first ideas I wrote in response. This should be considered a supplement to the official documentation on the Linux development process. The official documentation covers much more of the mechanical aspects and these principles I've written are more high level guidelines.

Colton's Kernel Development Guidelines

  1. Don't be intimidated. Linux is millions of lines, but most of it is well organized, easily separable, and has good docs. I work on KVM, mostly for ARM platforms, so I only care about 70k lines 90% of the time. That's things under virt/ and under arch/arm64/. The best way to avoid being overwhelmed is to figure out which functionality of the kernel you want to focus on and where it lives in the codebase.
  2. Make sure you have tooling to jump to definitions and find references to symbols. Make sure it works fast enough you use it often. For this reason I lean towards simpler tools like ctags/etags. In my experience, a lot of IDEs aren't built to handle codebases as big as Linux and their tooling tends to be grindingly slow even on simple tasks. (If the situation is better than that, please let me know).
  3. Read the documentation for your chosen subsystem under Documentation. It's capitalized specifically so it will show up first in a directory listing. There's a lot of good insight into the high level design and APIs. There is also a pretty web interface at https://docs.kernel.org. If you are ever questioning the how or why something is done the way it is, this should be your first stop.
  4. Remember the function of a kernel is to bridge the gap between userspace and hardware. It uses hardware resources to provide services to userspace. Pay particular attention and read the documentation for the userspace API (uAPI). These are the entry points into the kernel code. Once you know what service the kernel provides that you want to work on, narrowing it down to the specific uAPI functions and reading those will give you the best idea of the shape of the code.
  5. Don't be afraid to ask questions. If you get overwhelmed, especially if you are unsure about what you are looking at diving into new code, ask either a kernel developer you know or someone on the kernel mailing list.
  6. Subscribe to the narrowest possible kernel mailing list. The main mailing list is massive and it's easy to get lost in the noise. Check out this list of mailing lists. This is also a useful archive for searching these lists. The primary one I follow is kvmarm. You don't need to read it in depth, but skimming gives a good idea of current development hot spots and who the live players are. This is also good for understanding how the review process works and the etiquette (there are guides for this in the docs mentioned above too) .
  7. Build, build, build. The first line of defense for catching problems with your code is the compiler and upstream reviewers have very little patience if you send code that doesn't build. In the interest of that, develop on a powerful computer with lots of cores that makes building the complete kernel fast. Development velocity takes a big hit doing kernel development because of the speed of building and testing. Minimize that impact if at all possible.
  8. Debug virtually. As much as you can, get comfortable booting and debugging kernels with QEMU. Related to the above, development velocity is slow. Booting and testing on hardware is slow and it's much harder to recover machine state when you have a crash. This is another area where having a fast computer helps a ton.
  9. Use the existing tests. Learn about what is under the tools/ directory and specifically tools/testing, especially if it applies to your subsystem. If any tests are related to your changes, make sure they pass before and after your series. They encode a lot of valuable knowledge about different edge cases you have to handle.
  10. Develop a thick skin. OS work is hard and the mailing lists can be mean. These are often grumpy old developers from the other side of the world who don't have to look you in the face to tell you your code sucks. Taking from Linus's example, they tend to be extremely blunt because that is the clearest method of communication. Try not to take it personally.