Installing the Target Kernel

Grab the kernel source code and headers from github and the apt package manager

# update the packages
sudo apt update 
# Install the following packages so that all essential components for kernel
sudo apt install -y \
linux-image-5.15.0-136-generic \
linux-modules-5.15.0-136-generic \
linux-tools-5.15.0-136-generic
# If you also need header files—for compiling third-party modules, for example—add:
sudo apt install -y linux-headers-5.15.0-136-generic

Edit the GRUB file, so you can easily choose to which kernel version you want to boot in:

# Open the grub file
sudo nano /etc/default/grub

Change these following lines:

GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0

To be something like this:

GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10

Update the GRUB, and reboot.

sudo update-grub
sudo reboot

The Holy Symbols... Where Are You?

Option A: Adding Debugging Symbols to the Linux Kernel

You can add debug symbols via the Ubuntu's debug packages.

# enable ddebs if you haven't yet
sudo apt install -y ubuntu-dbgsym-keyring
echo "deb http://ddebs.ubuntu.com jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/ddebs.list
echo "deb http://ddebs.ubuntu.com jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt update

# install symbols for your exact kernel
sudo apt install -y linux-image-5.15.0-136-generic-dbgsym

Then use:

/usr/lib/debug/boot/vmlinux-5.15.0-136-generic

This file is unstripped with DWARF and perfect for GDB/KGDB.

Option B: Rebuild your custom kernel with symbols (unstripped)

1) Make sure the config enables debug info

From your kernel source tree (matching what you installed):

make scripts
scripts/config --enable  CONFIG_KGDB
scripts/config --enable  CONFIG_KGDB_SERIAL_CONSOLE
scripts/config --enable  CONFIG_GDB_SCRIPTS
scripts/config --enable  CONFIG_MAGIC_SYSRQ
scripts/config --enable  CONFIG_DEBUG_INFO
# choose a DWARF flavor (5 if toolchain supports it)
scripts/config --enable  CONFIG_DEBUG_INFO_DWARF5 || \
scripts/config --enable  CONFIG_DEBUG_INFO_DWARF4
# keep it simple (avoid split DWARF so all info is inside vmlinux)
scripts/config --disable CONFIG_DEBUG_INFO_SPLIT
yes "" | make olddefconfig

If you previously built without these, you must rebuild; stripped binaries cannot be “re-symbolized”.

2) Build (and keep the tree!)

Either build packages:

export LOCALVERSION=-kgdb
fakeroot make -j"$(nproc)" bindeb-pkg

or just build the kernel image in-tree:

make -j"$(nproc)"

Important: The unstripped file you want is in the build directory:

./vmlinux

Check it:

file vmlinux
# should NOT say "stripped"
readelf -S vmlinux | grep -E '\.debug_|\.symtab'

Now install the .deb you produced (if you used bindeb-pkg) but always keep a copy of this vmlinux somewhere safe; /boot/vmlinuz-* will still be compressed & stripped.

Manning KGDBoC on VMware

KGDBoC is way to debug the kernel with KGDB over a Serial Console, which is a perfect way to debug a Linux Desktop machine running on virtualization software (e.g. VMware).

Do this for both VMs’ Serial Port settings:

1. Use named pipes:

\\.\pipe\kgdb-vr

2. Endpoints:

3. On the debugger VM: This end is the client + The other end is a virtual machine

4. On the debuggee VM: This end is the server + The other end is a virtual machine

Then boot + attach

Power on the debuggee first (with kgdboc=ttyS0,115200 kgdbwait nokaslr in its GRUB cmdline).

Power on the debugger VM.

In the debugger VM:

sudo systemctl disable --now serial-getty@ttyS0.service 
# confirm which serial (ttyS0/ttyS1) (optional)
dmesg | grep -i ttyS
gdb -q /path/to/debuggee/vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyS0     # or /dev/ttyS1 if that’s what dmesg shows

Enable SysRq (for SysRq-g to break into kgdb)

To make the change for SysRq persistent:

echo 'kernel.sysrq=1' | sudo tee /etc/sysctl.d/99-sysrq.conf
sudo sysctl --system

Now you can trigger a break with:

echo g | sudo tee /proc/sysrq-trigger

GDB Helper Scripts

Linux source have some GDB helper scripts, which can ease the process of debugging your kernel.

1) Grab the scripts

Find the scripts and zip them from the source directory of your target machine.

cd ~/Desktop/dev/linux-5.15.0
zip -r scripts.zip scripts

2) Copy the Zip File

Grab the zip file from your target Linux to the host debugger Linux machine.

3) Import and Test

Inside the GDB interface:

python import sys; sys.path.append('/home/test/Desktop/scripts/gdb')
source scripts/gdb/vmlinux-gdb.py # No errors? Good sign!

And test if the scripts are working:

lx-ps

You should be presented with a similar output:

      TASK          PID    COMM
0xffffffff82e1b440  0x0  swapper/0
0xffff888100240000  0x1  systemd
0xffff888100244d40  0x2  kthreadd
0xffff888100243380  0x3  rcu_gp
0xffff8881002419c0  0x4  rcu_par_gp
0xffff888100253380  0x5  slub_flushwq
...
0xffff88810025b380  0xb  rcu_tasks_rude_
0xffff8881002599c0  0xc  rcu_tasks_trace
0xffff888100263380  0xd  ksoftirqd/0
0xffff8881002619c0  0xe  rcu_sched
0xffff888100260000  0xf  migration/0
0xffff888100264d40 0x10  idle_inject/0
0xffff888100aa4d40 0x12  cpuhp/0
...

All gdb commands at once (for faster debugging):

gdb vmlinux-5.15.0-136-DEBUG \
-ex "set serial baud 115200" \
-ex "target remote /dev/ttyS0" \
-ex "python import sys; sys.path.append('/home/test/Desktop/scripts/gdb')" \
-ex "source scripts/gdb/vmlinux-gdb.py" \
-ex "lx-cmdline"