summaryrefslogtreecommitdiff
path: root/Documentation/dev-tools
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2022-01-11 10:00:04 -0800
committerLinus Torvalds <torvalds@linux-foundation.org>2022-01-11 10:00:04 -0800
commit6f38be8f2ccd9babf04b9b23539108542a59fcb8 (patch)
treef1291c51111f9dc7ebc27e779dbd1dfc20a63b5b /Documentation/dev-tools
parent1be5bdf8cd5a194d981e65687367b0828c839c37 (diff)
parent87d6576ddf8ac25f36597bc93ca17f6628289c16 (diff)
downloadlinux-6f38be8f2ccd9babf04b9b23539108542a59fcb8.tar.gz
linux-6f38be8f2ccd9babf04b9b23539108542a59fcb8.tar.bz2
linux-6f38be8f2ccd9babf04b9b23539108542a59fcb8.zip
Merge tag 'docs-5.17' of git://git.lwn.net/linux
Pull documentation updates from Jonathan Corbet: "This isn't a hugely busy cycle for documentation, but a few significant things still showed up: - A documentation section for ARC processors - Reworked and enhanced KUnit documentation - The ability to pick your own theme for HTML builds; if the default "Read the Docs" theme isn't ugly enough for you, you can now pick an uglier one. - More Chinese translation work Plus the usual assortment of fixes and cleanups" * tag 'docs-5.17' of git://git.lwn.net/linux: (53 commits) scripts: sphinx-pre-install: Fix ctex support on Debian docs: discourage use of list tables docs: 5.Posting.rst: describe Fixes: and Link: tags Documentation: kgdb: Replace deprecated remotebaud docs: automarkup.py: Fix invalid HTML link output and broken URI fragments Documentation: refer to config RANDOMIZE_BASE for kernel address-space randomization Documentation: kgdb: properly capitalize the MAGIC_SYSRQ config docs/zh_CN: Update and fix a couple of typos scripts: sphinx-pre-install: add required ctex dependency Documentation: KUnit: Restyled Frequently Asked Questions Documentation: KUnit: Restyle Test Style and Nomenclature page Documentation: KUnit: Rework writing page to focus on writing tests Documentation: kunit: Reorganize documentation related to running tests Documentation: KUnit: Added KUnit Architecture Documentation: KUnit: Rewrite getting started Documentation: KUnit: Rewrite main page docs/zh_CN: Add zh_CN/accounting/delay-accounting.rst Documentation/sphinx: fix typos of "its" docs/zh_CN: Add sched-domains translation doc: fs: remove bdev_try_to_free_page related doc ...
Diffstat (limited to 'Documentation/dev-tools')
-rw-r--r--Documentation/dev-tools/index.rst1
-rw-r--r--Documentation/dev-tools/kgdb.rst6
-rw-r--r--Documentation/dev-tools/ktap.rst298
-rw-r--r--Documentation/dev-tools/kunit/architecture.rst204
-rw-r--r--Documentation/dev-tools/kunit/faq.rst73
-rw-r--r--Documentation/dev-tools/kunit/index.rst172
-rw-r--r--Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg81
-rw-r--r--Documentation/dev-tools/kunit/run_manual.rst57
-rw-r--r--Documentation/dev-tools/kunit/run_wrapper.rst247
-rw-r--r--Documentation/dev-tools/kunit/start.rst191
-rw-r--r--Documentation/dev-tools/kunit/style.rst105
-rw-r--r--Documentation/dev-tools/kunit/usage.rst578
12 files changed, 1427 insertions, 586 deletions
diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
index 010a2af1e7d9..4621eac290f4 100644
--- a/Documentation/dev-tools/index.rst
+++ b/Documentation/dev-tools/index.rst
@@ -32,6 +32,7 @@ Documentation/dev-tools/testing-overview.rst
kgdb
kselftest
kunit/index
+ ktap
.. only:: subproject and html
diff --git a/Documentation/dev-tools/kgdb.rst b/Documentation/dev-tools/kgdb.rst
index 43456244651a..f83ba2601e55 100644
--- a/Documentation/dev-tools/kgdb.rst
+++ b/Documentation/dev-tools/kgdb.rst
@@ -402,7 +402,7 @@ This is a quick example of how to use kdb.
2. Enter the kernel debugger manually or by waiting for an oops or
fault. There are several ways you can enter the kernel debugger
manually; all involve using the :kbd:`SysRq-G`, which means you must have
- enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config.
+ enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config.
- When logged in as root or with a super user session you can run::
@@ -461,7 +461,7 @@ This is a quick example of how to use kdb with a keyboard.
2. Enter the kernel debugger manually or by waiting for an oops or
fault. There are several ways you can enter the kernel debugger
manually; all involve using the :kbd:`SysRq-G`, which means you must have
- enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config.
+ enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config.
- When logged in as root or with a super user session you can run::
@@ -557,7 +557,7 @@ Connecting with gdb to a serial port
Example (using a directly connected port)::
% gdb ./vmlinux
- (gdb) set remotebaud 115200
+ (gdb) set serial baud 115200
(gdb) target remote /dev/ttyS0
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
new file mode 100644
index 000000000000..878530cb9c27
--- /dev/null
+++ b/Documentation/dev-tools/ktap.rst
@@ -0,0 +1,298 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========================================
+The Kernel Test Anything Protocol (KTAP)
+========================================
+
+TAP, or the Test Anything Protocol is a format for specifying test results used
+by a number of projects. It's website and specification are found at this `link
+<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test
+results. However, Kernel testing frameworks have special needs for test results
+which don't align with the original TAP specification. Thus, a "Kernel TAP"
+(KTAP) format is specified to extend and alter TAP to support these use-cases.
+This specification describes the generally accepted format of KTAP as it is
+currently used in the kernel.
+
+KTAP test results describe a series of tests (which may be nested: i.e., test
+can have subtests), each of which can contain both diagnostic data -- e.g., log
+lines -- and a final result. The test structure and results are
+machine-readable, whereas the diagnostic data is unstructured and is there to
+aid human debugging.
+
+KTAP output is built from four different types of lines:
+- Version lines
+- Plan lines
+- Test case result lines
+- Diagnostic lines
+
+In general, valid KTAP output should also form valid TAP output, but some
+information, in particular nested test results, may be lost. Also note that
+there is a stagnant draft specification for TAP14, KTAP diverges from this in
+a couple of places (notably the "Subtest" header), which are described where
+relevant later in this document.
+
+Version lines
+-------------
+
+All KTAP-formatted results begin with a "version line" which specifies which
+version of the (K)TAP standard the result is compliant with.
+
+For example:
+- "KTAP version 1"
+- "TAP version 13"
+- "TAP version 14"
+
+Note that, in KTAP, subtests also begin with a version line, which denotes the
+start of the nested test results. This differs from TAP14, which uses a
+separate "Subtest" line.
+
+While, going forward, "KTAP version 1" should be used by compliant tests, it
+is expected that most parsers and other tooling will accept the other versions
+listed here for compatibility with existing tests and frameworks.
+
+Plan lines
+----------
+
+A test plan provides the number of tests (or subtests) in the KTAP output.
+
+Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
+Plan lines follow version lines to indicate the number of nested tests.
+
+While there are cases where the number of tests is not known in advance -- in
+which case the test plan may be omitted -- it is strongly recommended one is
+present where possible.
+
+Test case result lines
+----------------------
+
+Test case result lines indicate the final status of a test.
+They are required and must have the format:
+
+.. code-block::
+
+ <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
+
+The result can be either "ok", which indicates the test case passed,
+or "not ok", which indicates that the test case failed.
+
+<number> represents the number of the test being performed. The first test must
+have the number 1 and the number then must increase by 1 for each additional
+subtest within the same test at the same nesting level.
+
+The description is a description of the test, generally the name of
+the test, and can be any string of words (can't include #). The
+description is optional, but recommended.
+
+The directive and any diagnostic data is optional. If either are present, they
+must follow a hash sign, "#".
+
+A directive is a keyword that indicates a different outcome for a test other
+than passed and failed. The directive is optional, and consists of a single
+keyword preceding the diagnostic data. In the event that a parser encounters
+a directive it doesn't support, it should fall back to the "ok" / "not ok"
+result.
+
+Currently accepted directives are:
+
+- "SKIP", which indicates a test was skipped (note the result of the test case
+ result line can be either "ok" or "not ok" if the SKIP directive is used)
+- "TODO", which indicates that a test is not expected to pass at the moment,
+ e.g. because the feature it is testing is known to be broken. While this
+ directive is inherited from TAP, its use in the kernel is discouraged.
+- "XFAIL", which indicates that a test is expected to fail. This is similar
+ to "TODO", above, and is used by some kselftest tests.
+- “TIMEOUT”, which indicates a test has timed out (note the result of the test
+ case result line should be “not ok” if the TIMEOUT directive is used)
+- “ERROR”, which indicates that the execution of a test has failed due to a
+ specific error that is included in the diagnostic data. (note the result of
+ the test case result line should be “not ok” if the ERROR directive is used)
+
+The diagnostic data is a plain-text field which contains any additional details
+about why this result was produced. This is typically an error message for ERROR
+or failed tests, or a description of missing dependencies for a SKIP result.
+
+The diagnostic data field is optional, and results which have neither a
+directive nor any diagnostic data do not need to include the "#" field
+separator.
+
+Example result lines include:
+
+.. code-block::
+
+ ok 1 test_case_name
+
+The test "test_case_name" passed.
+
+.. code-block::
+
+ not ok 1 test_case_name
+
+The test "test_case_name" failed.
+
+.. code-block::
+
+ ok 1 test # SKIP necessary dependency unavailable
+
+The test "test" was SKIPPED with the diagnostic message "necessary dependency
+unavailable".
+
+.. code-block::
+
+ not ok 1 test # TIMEOUT 30 seconds
+
+The test "test" timed out, with diagnostic data "30 seconds".
+
+.. code-block::
+
+ ok 5 check return code # rcode=0
+
+The test "check return code" passed, with additional diagnostic data “rcode=0”
+
+
+Diagnostic lines
+----------------
+
+If tests wish to output any further information, they should do so using
+"diagnostic lines". Diagnostic lines are optional, freeform text, and are
+often used to describe what is being tested and any intermediate results in
+more detail than the final result and diagnostic data line provides.
+
+Diagnostic lines are formatted as "# <diagnostic_description>", where the
+description can be any string. Diagnostic lines can be anywhere in the test
+output. As a rule, diagnostic lines regarding a test are directly before the
+test result line for that test.
+
+Note that most tools will treat unknown lines (see below) as diagnostic lines,
+even if they do not start with a "#": this is to capture any other useful
+kernel output which may help debug the test. It is nevertheless recommended
+that tests always prefix any diagnostic output they have with a "#" character.
+
+Unknown lines
+-------------
+
+There may be lines within KTAP output that do not follow the format of one of
+the four formats for lines described above. This is allowed, however, they will
+not influence the status of the tests.
+
+Nested tests
+------------
+
+In KTAP, tests can be nested. This is done by having a test include within its
+output an entire set of KTAP-formatted results. This can be used to categorize
+and group related tests, or to split out different results from the same test.
+
+The "parent" test's result should consist of all of its subtests' results,
+starting with another KTAP version line and test plan, and end with the overall
+result. If one of the subtests fail, for example, the parent test should also
+fail.
+
+Additionally, all result lines in a subtest should be indented. One level of
+indentation is two spaces: " ". The indentation should begin at the version
+line and should end before the parent test's result line.
+
+An example of a test with two nested subtests:
+
+.. code-block::
+
+ KTAP version 1
+ 1..1
+ KTAP version 1
+ 1..2
+ ok 1 test_1
+ not ok 2 test_2
+ # example failed
+ not ok 1 example
+
+An example format with multiple levels of nested testing:
+
+.. code-block::
+
+ KTAP version 1
+ 1..2
+ KTAP version 1
+ 1..2
+ KTAP version 1
+ 1..2
+ not ok 1 test_1
+ ok 2 test_2
+ not ok 1 test_3
+ ok 2 test_4 # SKIP
+ not ok 1 example_test_1
+ ok 2 example_test_2
+
+
+Major differences between TAP and KTAP
+--------------------------------------
+
+Note the major differences between the TAP and KTAP specification:
+- yaml and json are not recommended in diagnostic messages
+- TODO directive not recognized
+- KTAP allows for an arbitrary number of tests to be nested
+
+The TAP14 specification does permit nested tests, but instead of using another
+nested version line, uses a line of the form
+"Subtest: <name>" where <name> is the name of the parent test.
+
+Example KTAP output
+--------------------
+.. code-block::
+
+ KTAP version 1
+ 1..1
+ KTAP version 1
+ 1..3
+ KTAP version 1
+ 1..1
+ # test_1: initializing test_1
+ ok 1 test_1
+ ok 1 example_test_1
+ KTAP version 1
+ 1..2
+ ok 1 test_1 # SKIP test_1 skipped
+ ok 2 test_2
+ ok 2 example_test_2
+ KTAP version 1
+ 1..3
+ ok 1 test_1
+ # test_2: FAIL
+ not ok 2 test_2
+ ok 3 test_3 # SKIP test_3 skipped
+ not ok 3 example_test_3
+ not ok 1 main_test
+
+This output defines the following hierarchy:
+
+A single test called "main_test", which fails, and has three subtests:
+- "example_test_1", which passes, and has one subtest:
+
+ - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1"
+
+- "example_test_2", which passes, and has two subtests:
+
+ - "test_1", which is skipped, with the explanation "test_1 skipped"
+ - "test_2", which passes
+
+- "example_test_3", which fails, and has three subtests
+
+ - "test_1", which passes
+ - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.
+ - "test_3", which is skipped with the explanation "test_3 skipped"
+
+Note that the individual subtests with the same names do not conflict, as they
+are found in different parent tests. This output also exhibits some sensible
+rules for "bubbling up" test results: a test fails if any of its subtests fail.
+Skipped tests do not affect the result of the parent test (though it often
+makes sense for a test to be marked skipped if _all_ of its subtests have been
+skipped).
+
+See also:
+---------
+
+- The TAP specification:
+ https://testanything.org/tap-version-13-specification.html
+- The (stagnant) TAP version 14 specification:
+ https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
+- The kselftest documentation:
+ Documentation/dev-tools/kselftest.rst
+- The KUnit documentation:
+ Documentation/dev-tools/kunit/index.rst
diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst
new file mode 100644
index 000000000000..aa2cea821e25
--- /dev/null
+++ b/Documentation/dev-tools/kunit/architecture.rst
@@ -0,0 +1,204 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================
+KUnit Architecture
+==================
+
+The KUnit architecture can be divided into two parts:
+
+- Kernel testing library
+- kunit_tool (Command line test harness)
+
+In-Kernel Testing Framework
+===========================
+
+The kernel testing library supports KUnit tests written in C using
+KUnit. KUnit tests are kernel code. KUnit does several things:
+
+- Organizes tests
+- Reports test results
+- Provides test utilities
+
+Test Cases
+----------
+
+The fundamental unit in KUnit is the test case. The KUnit test cases are
+grouped into KUnit suites. A KUnit test case is a function with type
+signature ``void (*)(struct kunit *test)``.
+These test case functions are wrapped in a struct called
+``struct kunit_case``. For code, see:
+
+.. kernel-doc:: include/kunit/test.h
+ :identifiers: kunit_case
+
+.. note:
+ ``generate_params`` is optional for non-parameterized tests.
+
+Each KUnit test case gets a ``struct kunit`` context
+object passed to it that tracks a running test. The KUnit assertion
+macros and other KUnit utilities use the ``struct kunit`` context
+object. As an exception, there are two fields:
+
+- ``->priv``: The setup functions can use it to store arbitrary test
+ user data.
+
+- ``->param_value``: It contains the parameter value which can be
+ retrieved in the parameterized tests.
+
+Test Suites
+-----------
+
+A KUnit suite includes a collection of test cases. The KUnit suites
+are represented by the ``struct kunit_suite``. For example:
+
+.. code-block:: c
+
+ static struct kunit_case example_test_cases[] = {
+ KUNIT_CASE(example_test_foo),
+ KUNIT_CASE(example_test_bar),
+ KUNIT_CASE(example_test_baz),
+ {}
+ };
+
+ static struct kunit_suite example_test_suite = {
+ .name = "example",
+ .init = example_test_init,
+ .exit = example_test_exit,
+ .test_cases = example_test_cases,
+ };
+ kunit_test_suite(example_test_suite);
+
+In the above example, the test suite ``example_test_suite``, runs the
+test cases ``example_test_foo``, ``example_test_bar``, and
+``example_test_baz``. Before running the test, the ``example_test_init``
+is called and after running the test, ``example_test_exit`` is called.
+The ``kunit_test_suite(example_test_suite)`` registers the test suite
+with the KUnit test framework.
+
+Executor
+--------
+
+The KUnit executor can list and run built-in KUnit tests on boot.
+The Test suites are stored in a linker section
+called ``.kunit_test_suites``. For code, see:
+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/asm-generic/vmlinux.lds.h?h=v5.15#n945.
+The linker section consists of an array of pointers to
+``struct kunit_suite``, and is populated by the ``kunit_test_suites()``
+macro. To run all tests compiled into the kernel, the KUnit executor
+iterates over the linker section array.
+
+.. kernel-figure:: kunit_suitememorydiagram.svg
+ :alt: KUnit Suite Memory
+
+ KUnit Suite Memory Diagram
+
+On the kernel boot, the KUnit executor uses the start and end addresses
+of this section to iterate over and run all tests. For code, see:
+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/executor.c
+
+When built as a module, the ``kunit_test_suites()`` macro defines a
+``module_init()`` function, which runs all the tests in the compilation
+unit instead of utilizing the executor.
+
+In KUnit tests, some error classes do not affect other tests
+or parts of the kernel, each KUnit case executes in a separate thread
+context. For code, see:
+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/try-catch.c?h=v5.15#n58
+
+Assertion Macros
+----------------
+
+KUnit tests verify state using expectations/assertions.
+All expectations/assertions are formatted as:
+``KUNIT_{EXPECT|ASSERT}_<op>[_MSG](kunit, property[, message])``
+
+- ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an
+ expectation.
+
+ - For an expectation, if the check fails, marks the test as failed
+ and logs the failure.
+
+ - An assertion, on failure, causes the test case to terminate
+ immediately.
+
+ - Assertions call function:
+ ``void __noreturn kunit_abort(struct kunit *)``.
+
+ - ``kunit_abort`` calls function:
+ ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``.
+
+ - ``kunit_try_catch_throw`` calls function:
+ ``void complete_and_exit(struct completion *, long) __noreturn;``
+ and terminates the special thread context.
+
+- ``<op>`` denotes a check with options: ``TRUE`` (supplied property
+ has the boolean value “true”), ``EQ`` (two supplied properties are
+ equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not
+ contain an “err” value).
+
+- ``[_MSG]`` prints a custom message on failure.
+
+Test Result Reporting
+---------------------
+KUnit prints test results in KTAP format. KTAP is based on TAP14, see:
+https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md.
+KTAP (yet to be standardized format) works with KUnit and Kselftest.
+The KUnit executor prints KTAP results to dmesg, and debugfs
+(if configured).
+
+Parameterized Tests
+-------------------
+
+Each KUnit parameterized test is associated with a collection of
+parameters. The test is invoked multiple times, once for each parameter
+value and the parameter is stored in the ``param_value`` field.
+The test case includes a ``KUNIT_CASE_PARAM()`` macro that accepts a
+generator function.
+The generator function is passed the previous parameter and returns the next
+parameter. It also provides a macro to generate common-case generators based on
+arrays.
+
+For code, see:
+
+.. kernel-doc:: include/kunit/test.h
+ :identifiers: KUNIT_ARRAY_PARAM
+
+
+kunit_tool (Command Line Test Harness)
+======================================
+
+kunit_tool is a Python script ``(tools/testing/kunit/kunit.py)``
+that can be used to configure, build, exec, parse and run (runs other
+commands in order) test results. You can either run KUnit tests using
+kunit_tool or can include KUnit in kernel and parse manually.
+
+- ``configure`` command generates the kernel ``.config`` from a
+ ``.kunitconfig`` file (and any architecture-specific options).
+ For some architectures, additional config options are specified in the
+ ``qemu_config`` Python script
+ (For example: ``tools/testing/kunit/qemu_configs/powerpc.py``).
+ It parses both the existing ``.config`` and the ``.kunitconfig`` files
+ and ensures that ``.config`` is a superset of ``.kunitconfig``.
+ If this is not the case, it will combine the two and run
+ ``make olddefconfig`` to regenerate the ``.config`` file. It then
+ verifies that ``.config`` is now a superset. This checks if all
+ Kconfig dependencies are correctly specified in ``.kunitconfig``.
+ ``kunit_config.py`` includes the parsing Kconfigs code. The code which
+ runs ``make olddefconfig`` is a part of ``kunit_kernel.py``. You can
+ invoke this command via: ``./tools/testing/kunit/kunit.py config`` and
+ generate a ``.config`` file.
+- ``build`` runs ``make`` on the kernel tree with required options
+ (depends on the architecture and some options, for example: build_dir)
+ and reports any errors.
+ To build a KUnit kernel from the current ``.config``, you can use the
+ ``build`` argument: ``./tools/testing/kunit/kunit.py build``.
+- ``exec`` command executes kernel results either directly (using
+ User-mode Linux configuration), or via an emulator such
+ as QEMU. It reads results from the log via standard
+ output (stdout), and passes them to ``parse`` to be parsed.
+ If you already have built a kernel with built-in KUnit tests,
+ you can run the kernel and display the test results with the ``exec``
+ argument: ``./tools/testing/kunit/kunit.py exec``.
+- ``parse`` extracts the KTAP output from a kernel log, parses
+ the test results, and prints a summary. For failed tests, any
+ diagnostic output will be included.
diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst
index 5c6555d020f3..172e239791a8 100644
--- a/Documentation/dev-tools/kunit/faq.rst
+++ b/Documentation/dev-tools/kunit/faq.rst
@@ -4,56 +4,55 @@
Frequently Asked Questions
==========================
-How is this different from Autotest, kselftest, etc?
-====================================================
+How is this different from Autotest, kselftest, and so on?
+==========================================================
KUnit is a unit testing framework. Autotest, kselftest (and some others) are
not.
A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is supposed to
-test a single unit of code in isolation, hence the name. A unit test should be
-the finest granularity of testing and as such should allow all possible code
-paths to be tested in the code under test; this is only possible if the code
-under test is very small and does not have any external dependencies outside of
+test a single unit of code in isolation and hence the name *unit test*. A unit
+test should be the finest granularity of testing and should allow all possible
+code paths to be tested in the code under test. This is only possible if the
+code under test is small and does not have any external dependencies outside of
the test's control like hardware.
There are no testing frameworks currently available for the kernel that do not
-require installing the kernel on a test machine or in a VM and all require
-tests to be written in userspace and run on the kernel under test; this is true
-for Autotest, kselftest, and some others, disqualifying any of them from being
-considered unit testing frameworks.
+require installing the kernel on a test machine or in a virtual machine. All
+testing frameworks require tests to be written in userspace and run on the
+kernel under test. This is true for Autotest, kselftest, and some others,
+disqualifying any of them from being considered unit testing frameworks.
Does KUnit support running on architectures other than UML?
===========================================================
-Yes, well, mostly.
+Yes, mostly.
-For the most part, the KUnit core framework (what you use to write the tests)
-can compile to any architecture; it compiles like just another part of the
+For the most part, the KUnit core framework (what we use to write the tests)
+can compile to any architecture. It compiles like just another part of the
kernel and runs when the kernel boots, or when built as a module, when the
-module is loaded. However, there is some infrastructure,
-like the KUnit Wrapper (``tools/testing/kunit/kunit.py``) that does not support
-other architectures.
+module is loaded. However, there is infrastructure, like the KUnit Wrapper
+(``tools/testing/kunit/kunit.py``) that does not support other architectures.
-In short, this means that, yes, you can run KUnit on other architectures, but
-it might require more work than using KUnit on UML.
+In short, yes, you can run KUnit on other architectures, but it might require
+more work than using KUnit on UML.
For more information, see :ref:`kunit-on-non-uml`.
-What is the difference between a unit test and these other kinds of tests?
-==========================================================================
+What is the difference between a unit test and other kinds of tests?
+====================================================================
Most existing tests for the Linux kernel would be categorized as an integration
test, or an end-to-end test.
-- A unit test is supposed to test a single unit of code in isolation, hence the
- name. A unit test should be the finest granularity of testing and as such
- should allow all possible code paths to be tested in the code under test; this
- is only possible if the code under test is very small and does not have any
- external dependencies outside of the test's control like hardware.
+- A unit test is supposed to test a single unit of code in isolation. A unit
+ test should be the finest granularity of testing and, as such, allows all
+ possible code paths to be tested in the code under test. This is only possible
+ if the code under test is small and does not have any external dependencies
+ outside of the test's control like hardware.
- An integration test tests the interaction between a minimal set of components,
usually just two or three. For example, someone might write an integration
test to test the interaction between a driver and a piece of hardware, or to
test the interaction between the userspace libraries the kernel provides and
- the kernel itself; however, one of these tests would probably not test the
+ the kernel itself. However, one of these tests would probably not test the
entire kernel along with hardware interactions and interactions with the
userspace.
- An end-to-end test usually tests the entire system from the perspective of the
@@ -62,26 +61,26 @@ test, or an end-to-end test.
hardware with a production userspace and then trying to exercise some behavior
that depends on interactions between the hardware, the kernel, and userspace.
-KUnit isn't working, what should I do?
-======================================
+KUnit is not working, what should I do?
+=======================================
Unfortunately, there are a number of things which can break, but here are some
things to try.
-1. Try running ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output``
+1. Run ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output``
parameter. This might show details or error messages hidden by the kunit_tool
parser.
2. Instead of running ``kunit.py run``, try running ``kunit.py config``,
``kunit.py build``, and ``kunit.py exec`` independently. This can help track
down where an issue is occurring. (If you think the parser is at fault, you
- can run it manually against stdin or a file with ``kunit.py parse``.)
-3. Running the UML kernel directly can often reveal issues or error messages
- kunit_tool ignores. This should be as simple as running ``./vmlinux`` after
- building the UML kernel (e.g., by using ``kunit.py build``). Note that UML
- has some unusual requirements (such as the host having a tmpfs filesystem
- mounted), and has had issues in the past when built statically and the host
- has KASLR enabled. (On older host kernels, you may need to run ``setarch
- `uname -m` -R ./vmlinux`` to disable KASLR.)
+ can run it manually against ``stdin`` or a file with ``kunit.py parse``.)
+3. Running the UML kernel directly can often reveal issues or error messages,
+ ``kunit_tool`` ignores. This should be as simple as running ``./vmlinux``
+ after building the UML kernel (for example, by using ``kunit.py build``).
+ Note that UML has some unusual requirements (such as the host having a tmpfs
+ filesystem mounted), and has had issues in the past when built statically and
+ the host has KASLR enabled. (On older host kernels, you may need to run
+ ``setarch `uname -m` -R ./vmlinux`` to disable KASLR.)
4. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
(e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
around, so you can see what config was used after running ``kunit.py run``.
diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst
index 7af7dec83646..595205348d2d 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -1,13 +1,17 @@
.. SPDX-License-Identifier: GPL-2.0
-=========================================
-KUnit - Unit Testing for the Linux Kernel
-=========================================
+=================================
+KUnit - Linux Kernel Unit Testing
+=================================
.. toctree::
:maxdepth: 2
+ :caption: Contents:
start
+ architecture
+ run_wrapper
+ run_manual
usage
kunit-tool
api/index
@@ -16,82 +20,94 @@ KUnit - Unit Testing for the Linux Kernel
tips
running_tips
-What is KUnit?
-==============
-
-KUnit is a lightweight unit testing framework for the Linux kernel.
-
-KUnit is heavily inspired by JUnit, Python's unittest.mock, and
-Googletest/Googlemock for C++. KUnit provides facilities for defining unit test
-cases, grouping related test cases into test suites, providing common
-infrastructure for running tests, and much more.
-
-KUnit consists of a kernel component, which provides a set of macros for easily
-writing unit tests. Tests written against KUnit will run on kernel boot if
-built-in, or when loaded if built as a module. These tests write out results to
-the kernel log in `TAP <https://testanything.org/>`_ format.
-
-To make running these tests (and reading the results) easier, KUnit offers
-:doc:`kunit_tool <kunit-tool>`, which builds a `User Mode Linux
-<http://user-mode-linux.sourceforge.net>`_ kernel, runs it, and parses the test
-results. This provides a quick way of running KUnit tests during development,
-without requiring a virtual machine or separate hardware.
-
-Get started now: Documentation/dev-tools/kunit/start.rst
-
-Why KUnit?
-==========
-
-A unit test is supposed to test a single unit of code in isolation, hence the
-name. A unit test should be the finest granularity of testing and as such should
-allow all possible code paths to be tested in the code under test; this is only
-possible if the code under test is very small and does not have any external
-dependencies outside of the test's control like hardware.
-
-KUnit provides a common framework for unit tests within the kernel.
-
-KUnit tests can be run on most architectures, and most tests are architecture
-independent. All built-in KUnit tests run on kernel startup. Alternatively,
-KUnit and KUnit tests can be built as modules and tests will run when the test
-module is loaded.
-
-.. note::
-
- KUnit can also run tests without needing a virtual machine or actual
- hardware under User Mode Linux. User Mode Linux is a Linux architecture,
- like ARM or x86, which compiles the kernel as a Linux executable. KUnit
- can be used with UML either by building with ``ARCH=um`` (like any other
- architecture), or by using :doc:`kunit_tool <kunit-tool>`.
-
-KUnit is fast. Excluding build time, from invocation to completion KUnit can run
-several dozen tests in only 10 to 20 seconds; this might not sound like a big
-deal to some people, but having such fast and easy to run tests fundamentally
-changes the way you go about testing and even writing code in the first place.
-Linus himself said in his `git talk at Google
-<https://gist.github.com/lorn/1272686/revisions#diff-53c65572127855f1b003db4064a94573R874>`_:
-
- "... a lot of people seem to think that performance is about doing the
- same thing, just doing it faster, and that is not true. That is not what
- performance is all about. If you can do something really fast, really
- well, people will start using it differently."
-
-In this context Linus was talking about branching and merging,
-but this point also applies to testing. If your tests are slow, unreliable, are
-difficult to write, and require a special setup or special hardware to run,
-then you wait a lot longer to write tests, and you wait a lot longer to run