docs: new text on bisecting which also covers bug validation

Add a second document on bisecting regressions explaining the whole
process from beginning to end -- while also describing how to validate
if a problem is still present in mainline.  This "two in one" approach
is possible, as checking whenever a bug is in mainline is one of the
first steps before performing a bisection anyway and thus needs to be
described. Due to this approach the text also works quite nicely in
conjunction with Documentation/admin-guide/reporting-issues.rst, as it
covers all typical cases where users will need to build a kernel in
exactly the same order.

The text targets users that normally run kernels from their Linux
distributor who might never have compiled their own kernel.

This aim is why the first kernel built while following this guide is
generated from the latest mainline codebase. This will rule out that the
regression (a) was fixed already and (b) is caused by config change a
vendor distributor performed; checking mainline will furthermore (c)
determine if the issue is something that needs to be reported to the
regular developers or the stable team (this is needed even when readers
bisect within a stable series).

Only then are readers instructed to build their own variant of the
'good' kernel to validate the trimmed .config file created during early
in the guide, as performing a bisection with a broken one would be a
waste of time. There is a small downside of this order: readers might
have to go back to testing mainline, if it turns out there is a problem
with their .config. But that should be rare -- and if the regression was
already fixed readers might not get to this point anyway. Hence in the
end this order should mean that readers built less kernels overall.

This sequence allows the text to easily cover the "check if a bug is
present in the upstream kernel" case while only making things a tiny bit
more complicated.

The text tries to prevent readers from running into many mistakes users
are known to frequently make. The steps required for this might look
superfluous for people that are already familiar with bisections -- but
anyone with that knowledge should be able to adapt the instructions to
their use-case or will not need this text at all.

Style and structure of the text match the one
Documentation/admin-guide/quickly-build-trimmed-linux.rst uses. Quite a
few paragraphs are even copied from there and not changed at all or only
slightly. This will complicate maintenance, as some future changes to
one of these documents will have to be replicated in the other. But this
is the lesser evil: solutions like "sending readers from one document
over to the other" or "extracting the common parts into a separate
document" might work in other cases, but would be too confusing here
given the topic and the target audience.

Signed-off-by: Thorsten Leemhuis <linux@leemhuis.info>
[jc: Undo spurious removal of subsection header line]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <02b084a06de4ad61ac4ecd92b9265d4df4d03d71.1709282441.git.linux@leemhuis.info>

3 files changed, 1921 insertions, 0 deletions

diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index ed8a629e59c8..cb11d569b597 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -38,6 +38,7 @@ problems and bugs in particular.
    reporting-issues
    reporting-regressions
    quickly-build-trimmed-linux
+   verify-bugs-and-bisect-regressions
    bug-hunting
    bug-bisect
    tainted-kernels
diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
new file mode 100644
index 000000000000..54bde8bac95c
--- /dev/null
+++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
@@ -0,0 +1,1919 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
+.. [see the bottom of this file for redistribution information]
+
+=========================================
+How to verify bugs and bisect regressions
+=========================================
+
+This document describes how to check if some Linux kernel problem occurs in code
+currently supported by developers -- to then explain how to locate the change
+causing the issue, if it is a regression (e.g. did not happen with earlier
+versions).
+
+The text aims at people running kernels from mainstream Linux distributions on
+commodity hardware who want to report a kernel bug to the upstream Linux
+developers. Despite this intent, the instructions work just as well for users
+who are already familiar with building their own kernels: they help avoid
+mistakes occasionally made even by experienced developers.
+
+..
+   Note: if you see this note, you are reading the text's source file. You
+   might want to switch to a rendered version: it makes it a lot easier to
+   read and navigate this document -- especially when you want to look something
+   up in the reference section, then jump back to where you left off.
+..
+   Find the latest rendered version of this text here:
+   https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.rst.html
+
+The essence of the process (aka 'TL;DR')
+========================================
+
+*[If you are new to building or bisecting Linux, ignore this section and head
+over to the* ":ref:`step-by-step guide<introguide_bissbs>`" *below. It utilizes
+the same commands as this section while describing them in brief fashion. The
+steps are nevertheless easy to follow and together with accompanying entries
+in a reference section mention many alternatives, pitfalls, and additional
+aspects, all of which might be essential in your present case.]*
+
+**In case you want to check if a bug is present in code currently supported by
+developers**, execute just the *preparations* and *segment 1*; while doing so,
+consider the newest Linux kernel you regularly use to be the 'working' kernel.
+In the following example that's assumed to be 6.0.13, which is why the sources
+of v6.0 will be used to prepare the .config file.
+
+**In case you face a regression**, follow the steps at least till the end of
+*segment 2*. Then you can submit a preliminary report -- or continue with
+*segment 3*, which describes how to perform a bisection needed for a
+full-fledged regression report. In the following example 6.0.13 is assumed to be
+the 'working' kernel and 6.1.5 to be the first 'broken', which is why v6.0
+will be considered the 'good' release and used to prepare the .config file.
+
+* **Preparations**: set up everything to build your own kernels::
+
+    # * Remove any software that depends on externally maintained kernel modules
+    #   or builds any automatically during bootup.
+    # * Ensure Secure Boot permits booting self-compiled Linux kernels.
+    # * If you are not already running the 'working' kernel, reboot into it.
+    # * Install compilers and everything else needed for building Linux.
+    # * Ensure to have 15 Gigabyte free space in your home directory.
+    git clone -o mainline --no-checkout \
+      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
+    cd ~/linux/
+    git remote add -t master stable \
+      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+    git checkout --detach v6.0
+    # * Hint: if you used an existing clone, ensure no stale .config is around.
+    make olddefconfig
+    # * Ensure the former command picked the .config of the 'working' kernel.
+    # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up
+    #   VPNs, mount network shares, and briefly try the feature that is broken.
+    yes '' | make localmodconfig
+    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
+    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
+    # * Note, when short on storage space, check the guide for an alternative:
+    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
+      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
+    # * Hint: at this point you might want to adjust the build configuration;
+    #   you'll have to, if you are running Debian.
+    make olddefconfig
+    cp .config ~/kernel-config-working
+
+* **Segment 1**: build a kernel from the latest mainline codebase.
+
+  This among others checks if the problem was fixed already and which developers
+  later need to be told about the problem; in case of a regression, this rules
+  out a .config change as root of the problem.
+
+  a) Checking out latest mainline code::
+
+       cd ~/linux/
+       git checkout --force --detach mainline/master
+
+  b) Build, install, and boot a kernel::
+
+       cp ~/kernel-config-working .config
+       make olddefconfig
+       make -j $(nproc --all)
+       # * Make sure there is enough disk space to hold another kernel:
+       df -h /boot/ /lib/modules/
+       # * Note: on Arch Linux, its derivatives and a few other distributions
+       #   the following commands will do nothing at all or only part of the
+       #   job. See the step-by-step guide for further details.
+       command -v installkernel && sudo make modules_install install
+       # * Check how much space your self-built kernel actually needs, which
+       #   enables you to make better estimates later:
+       du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
+       du -sh /lib/modules/$(make -s kernelrelease)/
+       # * Hint: the output of the following command will help you pick the
+       #   right kernel from the boot menu:
+       make -s kernelrelease | tee -a ~/kernels-built
+       reboot
+       # * Once booted, ensure you are running the kernel you just built by
+       #   checking if the output of the next two commands matches:
+       tail -n 1 ~/kernels-built
+       uname -r
+
+  c) Check if the problem occurs with this kernel as well.
+
+* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.
+
+  This among others verifies the trimmed .config file actually works well, as
+  bisecting with it otherwise would be a waste of time:
+
+  a) Start by checking out the sources of the 'good' version::
+
+       cd ~/linux/
+       git checkout --force --detach v6.0
+
+  b) Build, install, and boot a kernel as described earlier in *segment 1,
+     section b* -- just feel free to skip the 'du' commands, as you have a rough
+     estimate already.
+
+  c) Ensure the feature that regressed with the 'broken' kernel actually works
+     with this one.
+
+* **Segment 3**: perform and validate the bisection.
+
+  a) In case your 'broken' version is a stable/longterm release, add the Git
+     branch holding it::
+
+       git remote set-branches --add stable linux-6.1.y
+       git fetch stable
+
+  b) Initialize the bisection::
+
+       cd ~/linux/
+       git bisect start
+       git bisect good v6.0
+       git bisect bad v6.1.5
+
+  c) Build, install, and boot a kernel as described earlier in *segment 1,
+     section b*.
+
+     In case building or booting the kernel fails for unrelated reasons, run
+     ``git bisect skip``. In all other outcomes, check if the regressed feature
+     works with the newly built kernel. If it does, tell Git by executing
+     ``git bisect good``; if it does not, run ``git bisect bad`` instead.
+
+     All three commands will make Git checkout another commit; then re-execute
+     this step (e.g. build, install, boot, and test a kernel to then tell Git
+     the outcome). Do so again and again until Git shows which commit broke
+     things. If you run short of disk space during this process, check the
+     "Supplementary tasks" section below.
+
+  d) Once your finished the bisection, put a few things away::
+
+       cd ~/linux/
+       git bisect log > ~/bisect-log
+       cp .config ~/bisection-config-culprit
+       git bisect reset
+
+  e) Try to verify the bisection result::
+
+       git checkout --force --detach mainline/master
+       git revert --no-edit cafec0cacaca0
+
+    This is optional, as some commits are impossible to revert. But if the
+    second command worked flawlessly, build, install, and boot one more kernel
+    kernel, which should not show the regression.
+
+* **Supplementary tasks**: cleanup during and after the process.
+
+  a) To avoid running out of disk space during a bisection, you might need to
+     remove some kernels you built earlier. You most likely want to keep those
+     you built during segment 1 and 2 around for a while, but you will most
+     likely no longer need kernels tested during the actual bisection
+     (Segment 3 c). You can list them in build order using::
+
+       ls -ltr /lib/modules/*-local*
+
+    To then for example erase a kernel that identifies itself as
+    '6.0-rc1-local-gcafec0cacaca0', use this::
+
+       sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0
+       sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0
+       # * Note, if kernel-install is missing, you will have to
+       #   manually remove the kernel image and related files.
+
+  b) If you performed a bisection and successfully validated the result, feel
+     free to remove all kernels built during the actual bisection (Segment 3 c);
+     the kernels you built earlier and later you might want to keep around for
+     a week or two.
+
+.. _introguide_bissbs:
+
+Step-by-step guide on how to verify bugs and bisect regressions
+===============================================================
+
+This guide describes how to set up your own Linux kernels for investigating bugs
+or regressions you intent to report. How far you want to follow the instructions
+depends on your issue:
+
+Execute all steps till the end of *segment 1* to **verify if your kernel problem
+is present in code supported by Linux kernel developers**. If it is, you are all
+set to report the bug -- unless it did not happen with earlier kernel versions,
+as then your want to at least continue with *segment 2* to **check if the issue
+qualifies as regression** which receive priority treatment. Depending on the
+outcome you then are ready to report a bug or submit a preliminary regression
+report; instead of the latter your could also head straight on and follow
+*segment 3* to **perform a bisection** for a full-fledged regression report
+developers are obliged to act upon.
+
+ :ref:`Preparations: set up everything to build your own kernels.<introprep_bissbs>`
+
+ :ref:`Segment 1: try to reproduce the problem with the latest codebase.<introlatestcheck_bissbs>`
+
+ :ref:`Segment 2: check if the kernels you build work fine.<introworkingcheck_bissbs>`
+
+ :ref:`Segment 3: perform a bisection and validate the result.<introbisect_bissbs>`
+
+ :ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`
+
+The steps in each segment illustrate the important aspects of the process, while
+a comprehensive reference section holds additional details. The latter sometimes
+also outlines alternative approaches, pitfalls, as well as problems that might
+occur at the particular step -- and how to get things rolling again.
+
+For further details on how to report Linux kernel issues or regressions check
+out Documentation/admin-guide/reporting-issues.rst, which works in conjunction
+with this document. It among others explains why you need to verify bugs with
+the latest 'mainline' kernel, even if you face a problem with a kernel from a
+'stable/longterm' series; for users facing a regression it also explains that
+sending a preliminary report after finishing segment 2 might be wise, as the
+regression and its culprit might be known already. For further details on
+what actually qualifies as a regression check out
+Documentation/admin-guide/reporting-regressions.rst.
+
+.. _introprep_bissbs:
+
+Preparations: set up everything to build your own kernels
+---------------------------------------------------------
+
+.. _backup_bissbs:
+
+* Create a fresh backup and put system repair and restore tools at hand, just
+  to be prepared for the unlikely case of something going sideways.
+
+  [:ref:`details<backup_bisref>`]
+
+.. _vanilla_bissbs:
+
+* Remove all software that depends on externally developed kernel drivers or
+  builds them automatically. That includes but is not limited to DKMS, openZFS,
+  VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).
+
+  [:ref:`details<vanilla_bisref>`]
+
+.. _secureboot_bissbs:
+
+* On platforms with 'Secure Boot' or similar solutions, prepare everything to
+  ensure the system will permit your self-compiled kernel to boot. The
+  quickest and easiest way to achieve this on commodity x86 systems is to
+  disable such techniques in the BIOS setup utility; alternatively, remove
+  their restrictions through a process initiated by
+  ``mokutil --disable-validation``.
+
+  [:ref:`details<secureboot_bisref>`]
+
+.. _rangecheck_bissbs:
+
+* Determine the kernel versions considered 'good' and 'bad' throughout this
+  guide.
+
+  Do you follow this guide to verify if a bug is present in the code developers
+  care for? Then consider the mainline release your 'working' kernel (the newest
+  one you regularly use) is based on to be the 'good' version; if your 'working'
+  kernel for example is '6.0.11', then your 'good' kernel is 'v6.0'.
+
+  In case you face a regression, it depends on the version range where the
+  regression was introduced:
+
+  * Something which used to work in Linux 6.0 broke when switching to Linux
+    6.1-rc1? Then henceforth regard 'v6.0' as the last known 'good' version
+    and 'v6.1-rc1' as the first 'bad' one.
+
+  * Some function stopped working when updating from 6.0.11 to 6.1.4? Then for
+    the time being consider 'v6.0' as the last 'good' version and 'v6.1.4' as
+    the 'bad' one. Note, at this point it is merely assumed that 6.0 is fine;
+    this assumption will be checked in segment 2.
+
+  * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? In
+    that case you want to bisect within a stable/longterm series: consider
+    'v6.0.11' as the last known 'good' version and 'v6.0.13' as the first 'bad'
+    one. Note, in this case you still want to compile and test a mainline kernel
+    as explained in segment 1: the outcome will determine if you need to report
+    your issue to the regular developers or the stable team.
+
+  *Note, do not confuse 'good' version with 'working' kernel; the latter term
+  throughout this guide will refer to the last kernel that has been working
+  fine.*
+
+  [:ref:`details<rangecheck_bisref>`]
+
+.. _bootworking_bissbs:
+
+* Boot into the 'working' kernel and briefly use the apparently broken feature.
+
+  [:ref:`details<bootworking_bisref>`]
+
+.. _diskspace_bissbs:
+
+* Ensure to have enough free space for building Linux. 15 Gigabyte in your home
+  directory should typically suffice. If you have less available, be sure to pay
+  attention to later steps about retrieving the Linux sources and handling of
+  debug symbols: both explain approaches reducing the amount of space, which
+  should allow you to master these tasks with about 4 Gigabytes free space.
+
+  [:ref:`details<diskspace_bisref>`]
+
+.. _buildrequires_bissbs:
+
+* Install all software required to build a Linux kernel. Often you will need:
+  'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
+  'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
+  reference section shows how to quickly install those on various popular Linux
+  distributions.
+
+  [:ref:`details<buildrequires_bisref>`]
+
+.. _sources_bissbs:
+
+* Retrieve the mainline Linux sources; then change into the directory holding
+  them, as all further commands in this guide are meant to be executed from
+  there.
+
+  *Note, the following describe how to retrieve the sources using a full
+  mainline clone, which downloads about 2,75 GByte as of early 2024. The*
+  :ref:`reference section describes two alternatives <sources_bisref>` *:
+  one downloads less than 500 MByte, the other works better with unreliable
+  internet connections.*
+
+  Execute the following command to retrieve a fresh mainline codebase::
+
+    git clone -o mainline --no-checkout \
+      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
+    cd ~/linux/
+
+  [:ref:`details<sources_bisref>`]
+
+.. _oldconfig_bissbs:
+
+* Start preparing a kernel build configuration (the '.config' file).
+
+  Before doing so, ensure you are still running the 'working' kernel an earlier
+  step told you to boot; if you are unsure, check the current kernel release
+  identifier using ``uname -r``.
+
+  Afterwards check out the source code for the version earlier established as
+  'good' and create a .config file::
+
+    git checkout --detach v6.0
+    make olddefconfig
+
+  The second command will try to locate the build configuration file for the
+  running kernel and then adjust it for the needs of the kernel sources you
+  checked out. While doing so, it will print a few lines you need to check.
+
+  Look out for a line starting with '# using defaults found in'. It should be
+  followed by a path to a file in '/boot/' that contains the release identifier
+  of your currently working kernel. If the line instead continues with something
+  like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find
+  the .config file for your running kernel -- in which case you have to put one
+  there manually, as explained in the reference section.
+
+  In case you can not find such a line, look for one containing '# configuration
+  written to .config'. If that's the case you have a stale build configuration
+  lying around. Unless you intend to use it, delete it; afterwards run
+  'make olddefconfig' again and check if it now picked up the right config file
+  as base.
+
+  [:ref:`details<oldconfig_bisref>`]
+
+.. _localmodconfig_bissbs:
+
+* Disable any kernel modules apparently superfluous for your setup. This is
+  optional, but especially wise for bisections, as it speeds up the build
+  process enormously -- at least unless the .config file picked up in the
+  previous step was already tailored to your and your hardware needs, in which
+  case you should skip this step.
+
+  To prepare the trimming, connect external hardware you occasionally use (USB
+  keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted
+  since you started that guide, ensure that you tried using the feature causing
+  trouble since you started the system. Only then trim your .config::
+
+     yes '' | make localmodconfig
+
+  There is a catch to this, as the 'apparently' in initial sentence of this step
+  and the preparation instructions already hinted at:
+
+  The 'localmodconfig' target easily disables kernel modules for features only
+  used occasionally -- like modules for external peripherals not yet connected
+  since booting, virtualization software not yet utilized, VPN tunnels, and a
+  few other things. That's because some tasks rely on kernel modules Linux only
+  loads when you execute tasks like the aforementioned ones for the first time.
+
+  This drawback of localmodconfig is nothing you should lose sleep over, but
+  something to keep in mind: if something is misbehaving with the kernels built
+  during this guide, this is most likely the reason. You can reduce or nearly
+  eliminate the risk with tricks outlined in the reference section; but when
+  building a kernel just for quick testing purposes this is usually not worth
+  spending much effort on, as long as it boots and allows to properly test the
+  feature that causes trouble.
+
+  [:ref:`details<localmodconfig_bisref>`]
+
+.. _tagging_bissbs:
+
+* Ensure all the kernels you will build are clearly identifiable using a special
+  tag and a unique version number::
+
+    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'
+    ./scripts/config -e CONFIG_LOCALVERSION_AUTO
+
+  [:ref:`details<tagging_bisref>`]
+
+.. _debugsymbols_bissbs:
+
+* Decide how to handle debug symbols.
+
+  In the context of this document it is often wise to enable them, as there is a
+  decent chance you will need to decode a stack trace from a 'panic', 'Oops',
+  'warning', or 'BUG'::
+
+    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \
+      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS
+
+  But if you are extremely short on storage space, you might want to disable
+  debug symbols instead::
+
+    ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
+      -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
+
+  [:ref:`details<debugsymbols_bisref>`]
+
+.. _configmods_bissbs:
+
+* Check if you may want or need to adjust some other kernel configuration
+  options:
+
+  * Are you running Debian? Then you want to avoid known problems by performing
+    additional adjustments explained in the reference section.
+
+    [:ref:`details<configmods_distros_bisref>`].
+
+  * If you want to influence other aspects of the configuration, do so now
+    by using make targets like 'menuconfig' or 'xconfig'.
+
+    [:ref:`details<configmods_individual_bisref>`].
+
+.. _saveconfig_bissbs:
+
+* Reprocess the .config after the latest adjustments and store it in a safe
+  place::
+
+     make olddefconfig
+     cp .config ~/kernel-config-working
+
+  [:ref:`details<saveconfig_bisref>`]
+
+.. _introlatestcheck_bissbs:
+
+Segment 1: try to reproduce the problem with the latest codebase
+----------------------------------------------------------------
+
+The following steps verify if the problem occurs with the code currently
+supported by developers. In case you face a regression, it also checks that the
+problem is not caused by some .config change, as reporting the issue then would
+be a waste of time. [:ref:`details<introlatestcheck_bisref>`]
+
+.. _checkoutmaster_bissbs:
+
+* Check out the latest Linux codebase::
+
+    cd ~/linux/
+    git checkout --force --detach mainline/master
+
+  [:ref:`details<checkoutmaster_bisref>`]
+
+.. _build_bissbs:
+
+* Build the image and the modules of your first kernel using the config file you
+  prepared::
+
+    cp ~/kernel-config-working .config
+    make olddefconfig
+    make -j $(nproc --all)
+
+  If you want your kernel packaged up as deb, rpm, or tar file, see the
+  reference section for alternatives, which obviously will require other
+  steps to install as well.
+
+  [:ref:`details<build_bisref>`]
+
+.. _install_bissbs:
+
+* Install your newly built kernel.
+
+  Before doing so, consider checking if there is still enough room for it::
+
+    df -h /boot/ /lib/modules/
+
+  150 MByte in /boot/ and 200 in /lib/modules/ usually suffice. Those are rough
+  estimates assuming the worst case. How much your kernels actually require will
+  be determined later.
+
+  Now install the kernel, which will be saved in parallel to the kernels from
+  your Linux distribution::
+
+    command -v installkernel && sudo make modules_install install
+
+  On many commodity Linux distributions this will take care of everything
+  required to boot your kernel. You might want to ensure that's the case by
+  checking if your boot loader's configuration was updated; furthermore ensure
+  an initramfs (also known as initrd) exists, which on many distributions can be
+  achieved by running ``ls -l /boot/init*$(make -s kernelrelease)*``. Those
+  steps are recommended, as there are quite a few Linux distribution where above
+  command is insufficient:
+
+  * On Arch Linux, its derivatives, many immutable Linux distributions, and a
+    few others the above command does nothing at, as they lack 'installkernel'
+    executable.
+
+  * Some distributions install the kernel, but don't add an entry for your
+    kernel in your boot loader's configuration -- the kernel thus won't show up
+    in the boot menu.
+
+  * Some distributions add a boot loader menu entry, but don't create an
+    initramfs on installation -- in that case your kernel most likely will be
+    unable to mount the root partition during bootup.
+
+  If any of that applies to you, see the reference section for further guidance.
+  Once you figured out what to do, consider writing down the necessary
+  installation steps: if you will build more kernels as described in
+  segment 2 and 3, you will have to execute these commands every time that
+  ``command -v installkernel [...]`` comes up again.
+
+  [:ref:`details<install_bisref>`]
+
+.. _storagespace_bissbs:
+
+* In case you plan to follow this guide further, check how much storage space
+  the kernel, its modules, and other related files like the initramfs consume::
+
+    du -ch /boot/*$(make -s kernelrelease)* | tail -n 1
+    du -sh /lib/modules/$(make -s kernelrelease)/
+
+  Write down or remember those two values for later: they enable you to prevent
+  running out of disk space accidentally during a bisection.
+
+  [:ref:`details<storagespace_bisref>`]
+
+.. _kernelrelease_bissbs:
+
+* Show and store the kernelrelease identifier of the kernel you just built::
+
+    make -s kernelrelease | tee -a ~/kernels-built
+
+  Remember the identifier momentarily, as it will help you pick the right kernel
+  from the boot menu upon restarting.
+
+.. _recheckbroken_bissbs:
+
+* Reboot into the kernel you just built and check if the feature that is
+  expected to be broken really is.
+
+  Start by making sure the kernel you booted is the one you just built. When
+  unsure, check if the output of these commands show the exact same release
+  identifier::
+
+    tail -n 1 ~/kernels-built
+    uname -r
+
+ Now verify if the feature that causes trouble works with your newly built
+ kernel. If things work while investigating a regression, check the reference
+ section for further details.
+
+  [:ref:`details<recheckbroken_bisref>`]
+
+.. _recheckstablebroken_bissbs:
+
+* Are you facing a problem within a stable/longterm release, but failed to
+  reproduce it with the mainline kernel you just built? Then check if the latest
+  codebase for the particular series might already fix the problem. To do so,
+  add the stable series Git branch for your 'good' kernel and check out the
+  latest version::
+
+    cd ~/linux/
+    git remote set-branches --add stable linux-6.0.y
+    git fetch stable
+    git checkout --force --detach linux-6.0.y
+
+  Now use the checked out code to build and install another kernel using the
+  commands the earlier steps already described in more detail::
+
+    cp ~/kernel-config-working .config
+    make olddefconfig
+    make -j $(nproc --all)
+    # * Check if the free space suffices holding another kernel:
+    df -h /boot/ /lib/modules/
+    command -v installkernel && sudo make modules_install install
+    make -s kernelrelease | tee -a ~/kernels-built
+    reboot
+
+  Now verify if you booted the kernel you intended to start, to then check if
+  everything works fine with this kernel::
+
+    tail -n 1 ~/kernels-built
+    uname -r
+
+  [:ref:`details<recheckstablebroken_bisref>`]
+
+Do you follow this guide to verify if a problem is present in the code
+currently supported by Linux kernel developers? Then you are done at this
+point. If you later want to remove the kernel you just built, check out
+:ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`.
+
+In case you face a regression, move on and execute at least the next segment
+as well.
+
+.. _introworkingcheck_bissbs:
+
+Segment 2: check if the kernels you build work fine
+---------------------------------------------------
+
+In case of a regression, you now want to ensure the trimmed configuration file
+you created earlier works as expected; a bisection with the .config file
+otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`]
+
+.. _recheckworking_bissbs:
+
+* Build your own variant of the 'working' kernel and check if the feature that
+  regressed works as expected with it.
+
+  Start by checking out the sources for the version earlier established as
+  'good'::
+
+    cd ~/linux/
+    git checkout --detach v6.0
+
+  Now use the checked out code to configure, build, and install another kernel
+  using the commands the previous subsection explained in more detail::
+
+    cp ~/kernel-config-working .config
+    make olddefconfig
+    make -j $(nproc --all)
+    # * Check if the free space suffices holding another kernel:
+    df -h /boot/ /lib/modules/
+    command -v installkernel && sudo make modules_install install
+    make -s kernelrelease | tee -a ~/kernels-built
+    reboot
+
+  When the system booted, you may want to verify once again that the
+  kernel you started is the one you just built:
+
+    tail -n 1 ~/kernels-built
+    uname -r
+
+  Now check if this kernel works as expected; if not, consult the reference
+  section for further instructions.
+
+  [:ref:`details<recheckworking_bisref>`]
+
+.. _introbisect_bissbs:
+
+Segment 3: perform the bisection and validate the result
+--------------------------------------------------------
+
+With all the preparations and precaution builds taken care of, you are now ready
+to begin the bisection. This will make you build quite a few kernels -- usually
+about 15 in case you encountered a regression when updating to a newer series
+(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build
+configuration created earlier this works a lot faster than many people assume:
+overall on average it will often just take about 10 to 15 minutes to compile
+each kernel on commodity x86 machines.
+
+* In case your 'bad' version is a stable/longterm release (say v6.1.5), add its
+  stable branch, unless you already did so earlier::
+
+    cd ~/linux/
+    git remote add -t master stable \
+      https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable.git
+    git remote set-branches --add stable linux-6.1.y
+    git fetch stable
+
+.. _bisectstart_bissbs:
+
+* Start the bisection and tell Git about the versions earlier established as
+  'good' and 'bad'::
+
+    cd ~/linux/
+    git bisect start
+    git bisect good v6.0
+    git bisect bad v6.1.5
+
+  [:ref:`details<bisectstart_bisref>`]
+
+.. _bisectbuild_bissbs:
+
+* Now use the code Git checked out to build, install, and boot a kernel using
+  the commands introduced earlier::
+
+    cp ~/kernel-config-working .config
+    make olddefconfig
+    make -j $(nproc --all)
+    # * Check if the free space suffices holding another kernel:
+    df -h /boot/ /lib/modules/
+    command -v installkernel && sudo make modules_install install
+    make -s kernelrelease | tee -a ~/kernels-built
+    reboot
+
+  If compilation fails for some reason, run ``git bisect skip`` and restart
+  executing the stack of commands from the beginning.
+
+  In case you skipped the "test latest codebase" step in the guide, check its
+  description as for why the 'df [...]' and 'make -s kernelrelease [...]'
+  commands are here.
+
+  Important note: the latter command from this point on will print release
+  identifiers that might look odd or wrong to you -- which they are not, as it's
+  totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'
+  if you bisect between versions 6.1 and 6.2 for example.
+
+  [:ref:`details<bisectbuild_bisref>`]
+
+.. _bisecttest_bissbs:
+
+* Now check if the feature that regressed works in the kernel you just built.
+
+  You again might want to start by making sure the kernel you booted is the one
+  you just built::
+
+    cd ~/linux/
+    tail -n 1 ~/kernels-built
+    uname -r
+
+  Now verify if the feature that regressed works at this kernel bisection point.
+  If it does, run this::
+
+    git bisect good
+
+  If it does not, run this::
+
+    git bisect bad
+
+  Be sure about what you tell Git, as getting this wrong just once will send the
+  rest of the bisection totally off course.
+
+  While the bisection is ongoing, Git will use the information you provided to
+  find and check out another bisection point for you to test. While doing so, it
+  will print something like 'Bisecting: 675 revisions left to test after this
+  (roughly 10 steps)' to indicate how many further changes it expects to be
+  tested. Now build and install another kernel using the instructions from the
+  previous step; afterwards follow the instructions in this step again.
+
+  Repeat this again and again until you finish the bisection -- that's the case
+  when Git after tagging a change as 'good' or 'bad' prints something like
+  'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right
+  afterwards it will show some details about the culprit including the patch
+  description of the change. The latter might fill your terminal screen, so you
+  might need to scroll up to see the message mentioning the culprit;
+  alternatively, run ``git bisect log > ~/bisection-log``.
+
+  [:ref:`details<bisecttest_bisref>`]
+
+.. _bisectlog_bissbs:
+
+* Store Git's bisection log and the current .config file in a safe place before
+  telling Git to reset the sources to the state before the bisection::
+
+    cd ~/linux/
+    git bisect log > ~/bisection-log
+    cp .config ~/bisection-config-culprit
+    git bisect reset
+
+  [:ref:`details<bisectlog_bisref>`]
+
+.. _revert_bissbs:
+
+* Try reverting the culprit on top of latest mainline to see if this fixes your
+  regression.
+
+  This is optional, as it might be impossible or hard to realize. The former is
+  the case, if the bisection determined a merge commit as the culprit; the
+  latter happens if other changes depend on the culprit. But if the revert
+  succeeds, it is worth building another kernel, as it validates the result of
+  a bisection, which can easily deroute; it furthermore will let kernel
+  developers know, if they can resolve the regression with a quick revert.
+
+  Begin by checking out the latest cod