Using Bisection to Debug Linux Kernel Configurations

Summary

This advanced guide illustrates how to debug kernel configurations. It is applicable when having one good configuration, and one desired configuration, and a wish to find out why the desired configuration is not working as intended.

Overview

For an embedded developer it now and then happens that one modifies the kernel configuration, and the system is no longer booting. Finding the offending CONFIG_ option can be quite a task, and this guide shows a procedure that can significantly reduce the loading.

The procedure is similar to a binary search or a git bisect. The idea is to split the good and bad configurations in the same place, and paste together a new config consisting of part good and part bad.

By testing a config where the top half is from the good config and bottom half is from the bad config, it is possible to deduce if a problem is in the top or bottom half.

Once this has been determined, one can keep the working half and make a new configuration with the problem half halved once more.While the theory sounds easyish, there are a few practicalities.

The good and bad configurations are of different length. Also it is sometimes not possible to cut and paste a config arbitrarily (there can be options that are dependent on others).Instead the cut point has to be selected roughly, and manually decide which CONFIG_ is a suitable
cutting point.

Details

To aid in this process a script is presented below:

#!/bin/sh
word=$1
if [ X$word = X ]
then
echo Need string
exit 1
fi
gpos=`grep -nr $word good | head -1 | cut -d ':' -f1`
bpos=`grep -nr $word bad | head -1 | cut -d ':' -f1`
blen=`wc -l bad | awk '{ print $1 }'`
head -${gpos} good > test
tail -`expr ${blen} - ${bpos}` bad >> test
echo config in test, line $gpos
cp test .config
make menuconfig

It assumes there are two master configurations in the working folder: good and bad. The names are hopefully self-explanatory. Note that these configs should be full configs and not minimized defconfigs.

Use the script by giving a CONFIG option to split by as argument. The resulting config will take all CONFIG options before the parameter option from the good config, and the remainder from the bad.

If the resulting config is working, one assume all CONFIGs below the parameter config are good, and instead make a new cutting point earlier in the file. And if the resulting config is bad, one should make the next cut later in the file.

A useful way of thinking is to keep track of the line numbers in the bad config that are known good as an interval [1, 3855] for example. And with each bisection update what knowledge have been gained.