Brendan Higgins recently proposed adding unit tests to the Linux kernel,
supplementing other development infrastructure such as
perf, autotest and
kselftest. The whole issue of testing is very dear to kernel developers’
hearts, because Linux sits at the core of the system and often has a very
strong stability/security requirement. Hosts of automated tests regularly
churn through kernel source code, reporting any oddities to the mailing
list.
Unit tests, Brendan said, specialize in testing standalone code snippets.
It was not necessary to run a whole kernel, or even to compile the kernel
source tree, in order to perform unit tests. The code to be tested could be
completely extracted from the tree and tested independently. Among other
benefits, this meant that dozens of unit tests could be performed in less
than a second, he explained.
Giving credit where credit was due, Brendan identified
JUnit, Python’s
unittest.mock and Googletest/Googlemock
for C++ as the inspirations for
this new KUnit testing idea.
Brendan also pointed out that since all code being unit-tested is
standalone and has no dependencies, this meant the tests also
were deterministic. Unlike on a running Linux system, where any number of pieces
of the running system might be responsible for a given problem, unit tests
would identify problem code with repeatable certainty.
Daniel Vetter replied extremely enthusiastically to Brendan’s work. In
particular, he said, “Having proper and standardized infrastructure for
kernel unit tests sounds terrific. In other words: I want.” He added that
he and some others already had been working on a much more specialized set
of unit tests for the Direct Rendering Manager (DRM) driver. Brendan’s
approach, he said, would be much more convenient than his own more
localized efforts.
Dan Williams was also very excited about Brendan’s work,
and he said he had
been doing a half-way job of unit tests on the libnvdimm (non-volatile
device) project code. He felt Brendan’s work was much more general-purpose,
and he wanted to convert his own tests to use KUnit.
Tim Bird replied to Brendan’s initial email as well, saying he thought unit
tests could be useful, but he wanted to make sure the behaviors were
correct. In particular, he wanted clarification on just how it was possible
to test standalone code. If the code were to be compiled independently,
would it then run on the local system? What if the local system had a
different hardware architecture from the system for which the code was
intended?
Also, who would maintain unit tests, and where would the tests live, within
the source tree? Would they clutter up the directory being tested, or would
they live
far away in a special directory reserved for test code? And finally, would
test code be easier to write than the code being tested? In other words,
could new developers cut their teeth on a project by writing test code, as
a gateway to helping work on a given driver or subsystem? Or would unit
tests have to be written by people who had total expertise in the area
already?
Brendan attempted to address each of those issues in turn. To start, he
confirmed that the test code was indeed extracted and compiled on the local
system. Eventually, he said, each test would compile into its own
completely independent test binary, although for the moment, they were all
lumped together into a single user-mode-linux (UML) binary.
In terms of cross-compiling test code for other architectures, Brendan felt
this would be hard to maintain and had decided not to support it. Tests
would run locally and would not depend on architecture-specific
characteristics.
In terms of where the unit tests would live, Brendan said they would be in
the same directory as the code being tested. So every directory would have
its own set of unit tests readily available and visible. The same person
maintaining the code being tested would maintain the tests themselves. The
unit tests, essentially, would become an additional element of every
project. That maintainer would then presumably require that all patches to
that driver or subsystem pass all the unit tests before they could be
accepted into the tree.
In terms of who was qualified to write unit tests for a given project,
Brendan explained:
In order to write a unit test, the person who writes
the test must understand what the code they are testing is supposed to do.
To some extent that will probably require someone with some expertise to
ensure that the test makes sense, and indeed a change that breaks a test
should be accompanied by an update to the test. On the other hand, I think
understanding what pre-existing code does and is supposed to do is much
easier than writing new code from scratch, and probably doesn’t require too
much expertise.
Brendan added that unit tests would probably reduce, rather than increase,
a maintainer’s workload. In spite of representing more code overall:
Code
with unit tests is usually cleaner, the tests tell me exactly what the code
is supposed to do, and I can run the tests (or ideally have an automated
service run the tests) that tell me that the code actually does what the
tests say it should. Even when it comes to writing code, I find that writing
code with unit tests ends up saving me time.
Overall, Brendan was very pleased by all the positive interest, and said he
planned to do additional releases to address the various technical
suggestions that came up during the course of discussion.
No voices really were raised in opposition to any of Brendan’s ideas. It
appears that unit tests may soon become a standard part of many drivers and
subsystems.
Note: if you’re mentioned above and want to post a response above the comment section, send a message with your response text to ljeditor@linuxjournal.com.