The Linux kernel is one of the most popular operating system kernels in the world. Less than 30 years after its humble beginnings in 1991, the Linux kernel now underpins modern computing infrastructure, with 2019 estimates of the number of running Linux kernels ranging upwards of twenty billion. To put that in perspective: There are about 3 Linux kernels for every living person.
The Linux kernel powers household appliances, smartphones, industrial automation, Internet data centers, almost all of the cloud, financial services, and supercomputers. It even powers a few percent of the world’s desktop systems, including the one that I am typing these words into. But the year of the Linux desktop continues its decades-long tradition of being next year.
But it wasn’t always that way.
A brave choice, big commitment in Linux’s early days
The Linux kernel was still a brave choice when IBM joined the Linux community in the late 1990s. IBM began its Linux-kernel work with a skunkworks port of Linux to the IBM mainframe and a corporate commitment that resulted in IBM’s investing $1B on Linux in 2001. Linux was ported to all IBM servers, and even to IBM Research’s Linux-powered wristwatch. Linux soon enjoyed widespread use within IBM’s hardware, software, and services.
Of course, IBM wasn’t the only player betting on Linux. For example, an IBM sales team spent much time preparing to convince a long-standing, technically conservative client to start moving towards Linux. When the team went to give their pitch, the client opened the discussion with: “We have decided that we are going with Linux. Will you be coming with us?” Although this destroyed untold hours of preparation, it produced a result beyond the sales team’s wildest imaginations.
And it wasn’t an isolated incident.
Setting Linux up for success
This widespread enthusiasm motivated IBM not only to make substantial contributions to Linux, but also to come to its defense. First, we committed to not attack Linux in the form of patent pledges. We took it a step further and opted to co-found the Open Invention Network, which helped defend open source projects such as the Linux kernel against attacks by patent holders. We made numerous visits to the courtroom to defend ourselves against a lawsuit related to our Linux involvement, and co-founded several umbrella organizations to facilitate open source projects, perhaps most notably helping to found the Linux Foundation.
IBM is also a strong technical contributor to the Linux kernel, ranking in the top ten corporate contributors and having maintainers for a wide range of Linux-kernel subsystems. Of course, IBM contributes heavily to support its own offerings, but it is also a strong contributor in the areas of scalability, robustness, security, and other areas that benefit the Linux ecosystem.
Of course, not everything that IBM attempted worked out. IBM’s scalability work in the scheduler was never accepted into the Linux kernel. Although its journaling filesystem (JFS) was accepted and remains part of the Linux kernel, it seems safe to say that JFS never achieved the level of popularity that IBM had hoped for. Nevertheless, it seems likely that IBM’s efforts helped to inspire the work leading to the Linux kernel’s excellent scalability, features, and functionality in its filesystems and scheduler.
In addition, these experiences taught IBM to work more closely with the community, paving the way to later substantial contributions. One example is the CPU-groups feature of the community’s scheduler that now underpins containers technologies such as Docker, along with the virtio feature that plays into the more traditional hypervisor-based virtualization. Another example is numerous improvements leading up to the community’s EXT4 filesystem. A final example is the device-tree hardware specification feature, originally developed for IBM’s Power servers but now also used by many embedded Linux systems.
Achieving impossible results
It has also been a great privilege for IBM to be involved in a number of Linux-kernel efforts that produced results widely believed to be impossible.
First, at the time that IBM joined the Linux kernel community, the kernel could scale to perhaps two or four CPUs. At the time there was a large patchset from SGI that permitted much higher scalability, but this patchset primarily addressed HPC workloads. About ten years of hard work across the community changed this situation dramatically, so that, despite the naysayers, the same Linux-kernel source code supports both deep embedded systems and huge servers with more than one thousand CPUs and terabytes of memory.
Second, it was once common knowledge that achieving sub-millisecond response times required a special-purpose, real-time operating system. In other words, sub-millisecond response times certainly could not be achieved by a general-purpose operating system such as the Linux kernel. IBM was an important part of a broad community effort that proved this to be wrong, as part of an award-winning effort including Raytheon and the US Navy. Although the real-time patchsett has not yet been fully integrated into the mainline Linux kernel, it does achieve not merely deep sub-millisecond response times, but rather deep sub-hundred-microsecond response times. And these response times are achieved not only on single-CPU embedded system, but also on systems with thousands of CPUs.
Third, only about a decade ago, it was common knowledge that battery-powered embedded systems required special-purpose operating systems. You might be surprised that IBM would be involved in kernel work in support of such systems. One reason for IBM’s involvement was that some of the same code that improves battery lifetime also improves the Linux kernel’s virtualization capabilities — capabilities important to the IBM mainframe. A second reason for IBM’s involvement was the large volume of ARM chips then produced by its semiconductor technology partners. This latter reason motivated IBM to cofound the Linaro consortium, which improved Linux support for ARM’s processor families. The result, as billions of Android smartphone users can attest, is that the Linux kernel has added battery-powered systems to its repertoire.
Fourth and finally, version 5.2 of the Linux kernel comprises 13,600 changes from 1,716 kernel developers. The vast majority of these changes were applied during the two-week merge window immediately following the release of version 5.1, with only a few hundred new changes appearing in each of the release candidates that appear at the end of each weekend following the merge window. This represents a huge volume of changes from a great many contributors, and with little formal coordination. Validating these changes is both a challenge and a first-class concern.
One of IBM’s contributions to validation is “-next” integration testing, which checks for conflicts among the contributions intended for the next merge window. The effects of -next integration testing, when combined with a number of other much-appreciated efforts throughout the community, has not been subtle. Ten years ago, serious kernel testing had to wait for the third or fourth release candidate due to bugs introduced during the preceding merge window. Today, serious kernel testing can almost always proceed with the first release candidate that comes out immediately at the close of the merge window.
But is Linux done yet?
A continuing effort
Although developers should be proud of the great increases in stability of the Linux kernel over the years, the kernel still has bugs, some of which are exploitable. There is a wide range of possible improvements from more aggressively applying tried testing techniques to over-the-horizon research topics such as formal verification. In addition, existing techniques are finding new applications, so that CPU hotplug (which was spearheaded by IBM in the early 2000s) has recently been used to mitigate hardware side-channel attack vectors.
The size of hardware systems is still increasing, which will require additional work on scalability. Many of these larger systems will be used in various cloud-computing environments, some of which will pose new mixed-workload challenges. Changing hardware devices, including accelerators and non-volatile memory, will require additional support from Linux, as well as from hardware features such as IBM’s Power Systems servers’ support of the NVLink and CAPI interconnects.
Finally, there is more to security than simply fixing bugs faster than attackers can exploit them (though that would be challenge enough!). Although there is a great deal of security work needed in a great many areas, one important advance is Pervasive Encryption for IBM Z.
IBM congratulates the Linux kernel community on its excellent progress over the decades, and looks forward to being part of future efforts overturning yet more morsels of common wisdom!