With Linux, You Don't Get One Kernel of Truth... You Get Many

As much as I love to poke at the inner workings of my computer, I'll admit that until recently, I didn't give much thought to which version of the Linux kernel my desktop system was running.

For most desktop users, this isn't all that odd. Compatibility of kernel modules is often critical for servers and production systems, but day-to-day desktop usage doesn't change much from update to update.

Two things motivated me to scrutinize the kernel version more closely: considerations for specific hardware; and a very scary bug recently identified in the Ubuntu distribution's latest release.

Having picked up a lot of useful tips in exploring different kernel versions, I decided to share what I've learned so far.

Rowing Down the Kernel Stream

Before diving in, it will help to get an idea of how the various distributions handle the kernel. The source of all the kernels that are repackaged and included in all the Linux distributions is the kernel produced by the Linux kernel development team.

This initiative, relative to the distributions based on it, is referred to as "upstream," because any team wishing to integrate the Linux kernel into its project (relationally "downstream") first inherits the base of code from the source kernel before choosing what to keep, modify, or remove -- it's like a river delta inheriting the debris washed down from upstream.

The template kernel produced by the upstream kernel team generally has the highest version number of any released kernel variant.

As the numbering scheme implies, though, not every downstream team starts tweaking the source kernel from the same, most recent point. If all of them did, kernel names simply could denote the distribution, so that, taking the present 4.14 kernel as the base, Debian would release its kernel under the name "linux-4.14-debian" and Arch Linux would call its kernel "linux-4.14-arch," etc.

Instead, depending on the goals a distribution hopes to accomplish and the experience its developers want users to have, each distribution takes a different upstream kernel version as its template and works off that, diverging to develop roughly in parallel with the upstream project.

For example, consider a fictional distribution, "X," based on kernel 4.10. When upstream Linux reached version 4.14.1, X might be at 4.10.1, and when upstream later advanced to 4.15, X might be at 4.10.8-5. Notice that no matter how high the upstream version goes, X never would go "above" version 4.10.x.

To show just how widely the distributions can deviate from their progenitor, here are a few kernels for prominent distributions as of their latest downloadable release (not their most recent update from an installed system):

• One of the more dated versions out in the wild is the one running on Ubuntu 16.04 LTS (for "Long Term Support") released all the way back in April 2016, which uses a customized version of 4.4.
• Debian 9, on its "Stable" release track, is running a variant of version 4.9.
• Linux Mint 18.3, the newest offering from a distribution that prides itself on stability and usability, ships with a tweak of version 4.10.
• On the more cutting-edge side, Ubuntu 17.10, released a few months back, includes its spin on kernel 4.13 by default.
• And for those seeking to stay hot on the heels of the upstream project, Arch Linux runs on 4.14, only a little behind the upstream's 4.15.

So what exactly are the differences between all these? As noted earlier, many of them lie outside the interest of most desktop users, as it's rare to find a user who actively uses more than a fraction of new modules (especially as some enable very niche hardware). That said, following are two examples of crucial kernel components that users would be wise to consider when choosing which version to run.

Stay Close - but Not Too Close

Intel recently revamped the design of its processors with the release of the Kaby Lake line, creating quite a powerhouse as a result.

While Windows has been able to harness its horsepower right out of the gate, it took Linux a little while to get a handle on it. To be precise, version 4.10 debuted the optimizations for Kaby Lake processors. These chips certainly will run on kernels from 4.9 and earlier, but users won't get the most out of their hardware, and may even wear it out faster than if they took advantage of the updated kernel.

This is just one instance of when it pays not to lag too far behind the upstream kernel, as you can spare newer systems' internals a lot of unnecessary wear and tear.

Still, there can be serious drawbacks to getting too close to the vanguard that is the source kernel, or too experimental. In a particularly unsettling example of this, Ubuntu users have encountered a severe bug in how Ubuntu's more recent kernel interacts with UEFI firmware, which runs the boot sequence on modern systems.

In the variant of the 4.13 kernel shipped with Ubuntu 17.10, Ubuntu activated an unfinished kernel module for Intel SPI devices, which incorrectly accessed and damaged the UEFI firmware in some computers, rendering UEFI unable to save settings changes or to boot from USB flash drives.

This bug is an apt demonstration of the relationship between upstream, downstream, and other parallel downstream flows. The bug seems to have been included in Ubuntu's kernel erroneously, by Ubuntu itself. Other distributions downstream from the source kernel project, running versions 4.13 or later -- such as Arch Linux and openSuse Tumbleweed -- appear not to be affected.

Linux distributions also can have upstream/downstream relationships to one another: Ubuntu is downstream from Debian, which in turn is directly downstream from the source kernel. However, since Ubuntu included the offending module, even the advanced Testing track of Debian on version 4.14 doesn't appear to be affected.

Linux Mint, on the other hand, is downstream from Ubuntu, but because its kernel -- branched from 4.10 -- is behind Ubuntu's, it doesn't look like it's affected either.

Although the version of the Linux kernel running or the features it includes aren't always immediately apparent to desktop users, there are times when these specifics can make your life easier -- or a lot harder.

If you really want to pore over all the details of each kernel, changelogs published on every update are your best bet. If you want to survey the offerings from the major distributions, poking around user forums is the way to go.

The only way to find out is to give it try. Between the primer upstream and the rich ecosystem of downstreams, you might be surprised at just how much the Linux kernel can do.