Being able to use LLDB to debug anything on my Mac has been the basis of my job for the last few years. Regardless of the particular use case – to debug my own program in Xcode or attach to another process – being able to set up breakpoints and inspect a program call stack frames and memory at runtime is just invaluable.

However, there’s one case where one cannot easily attach a debugger at any time: the kernel. Even if you could attach to the kernel on your local machine and halt its execution while you’re having a look around, the debugger wouldn’t be able to do much since it relies on the kernel to do, well anything. An alternative approach would be to use printf or IOLog and call it a day but that’s just not how I roll. If I need to debug something, I want a debugger.

In this article, we will discuss how one can set up a VMware Fusion virtual machine (VM) and use LLDB to do remote kernel debugging. Note that this approach could be used to debug any process remotely – which can be very useful for system critical processes such as launchd – but in this article we will mostly focus on the XNU kernel.

Setup

Setting up your environment for kernel debugging isn’t too hard but it can seem daunting at first since not much documentation is available.

In the old days, one would have needed to use two physical machines connected with an ethernet or firewire cable. However, nowadays one can easily install OS X on a VMware Fusion virtual machine. Since using multiple OS X systems on a single machine is pretty cool, we will setup the VM and discuss how to connect the debugger from the host.

I will assume that you are using VMware Fusion 7 and OS X 10.10.5. All this should very likely work with previous and later versions of VMware Fusion (or other virtualization products such as Parallels or VirtualBox, though I’m not sure since I’ve never used them) and OS X. It’s important to note however that El Capitan introduces System Integrity Protection that makes some breaking changes with respect to the boot-args arguments. I will update this article when El Capitan ships and the updated approach is final.

On the virtual machine

Install OS X 10.10

You will need to install the same version of OS X on the host system and the virtual machine. This is not technically required but it will make things easier. I’m currently using 10.10.5 but any version of Yosemite should work in the same way.

Install the Kernel Debug Kit

You should install the Kernel Debug Kit (KDK) from Apple Developer Downloads page. Make sure to pick the KDK that matches the version of OS X that you’re installed. The kit is a simple package install that you should run. It will install some components under /Library/Developer/KDKs.

If you take a look through this directory after installing, you should see something like this:

/Library/Developer/KDKs
    KDK_10.10.5_14F27.kdk
        ReadMe.html
        System
            Library
                Extensions
                    AppleUSBAudio.kext
                    AppleUSBAudio.kext.dSYM
                    ...
                Kernels
                    kernel.debug
                    kernel.debug.dSYM
                    ...

You now have a copy of the kernel itself and each kernel extension that Apple ships with the system, including the symbols for each of these executables. There is also a development and debug versions of the kernel that we will use in the VM.

The ReadMe.html file also has a ton of information about the KDK so make sure you at least skim through.

You will need the KDK to be installed on both the host and the VM:

It is needed on the VM to install the development version of the kernel
It is needed on the host in order for the debugger to use as the target or simply resolve the symbols

Install the development kernel

On OS X 10.10, the kernel executable is located under /Systems/Library/Kernels.

The KDK comes with a development and debug version of the kernel. To quote the KDK ReadMe.html docs:

The OS X Yosemite Kernel Debug Kit includes the DEVELOPMENT and DEBUG kernel builds. These both have additional assertions and error checking compared to the RELEASE kernel. The DEVELOPMENT kernel can be used for every-day use and has minimal performance overhead, while the DEBUG kernel has much more error checking.

It’s not technically required to install and boot into the development kernel to do kernel debugging (since the KDK has the symbols for the shipping kernel too and, as far as I can tell, the release kernel ships with the kdp_* symbols required for remote debugging) but we could definitely use the additional assertions and error checking so we will use it.

In order to install a distinct build of the kernel, there are two steps to take. We will discuss the boot-args changes in a following section so for now, we only need to copy the kernel in the right location.

To do so, you will need to copy the kernel.development executable located under the KDK install directory to /Systems/Library/Kernels:

# `KDK_10.10.5_14F27.kdk` is your current KDK install
damien$ sudo cp /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development /Systems/Library/Kernels

With this in place, your system can boot into the development kernel, assuming it’s told how to do so.

Update the NVRAM `boot-args`

The non-volatile random-access memory (NVRAM) is memory that retains its information when power is turned off. In particular, it can contain boot arguments that the system checks when booting. On OS X, its variables can be queried and manipulated by using the nvram command line utility. Note that this is drastically changing with OS X El Capitan but as far as Yosemite is concerned, nvram is the right tool to change boot arguments.

In order to set up our VM for debugging we will need to update the boot arguments by running the following command:

damien$ sudo nvram boot-args="debug=0x141 kext-dev-mode=1 kcsuffix=development pmuflags=1 -v"

where each boot argument does a very specific thing:

debug=0x141 enables the debug mode and sets a few debug options such as ensuring that the system on the VM waits for a remote debugger connection when booting. The Kernel Programming Guide has more info about the various options. Here we are using (DB_HALT | DB_ARP | DB_LOG_PI_SCRN).
kext-dev-mode=1 allows us to load unsigned kexts. This is not required if you’re only tweaking with the kernel itself but if you’re developing a kernel extension and don’t have a kext-enabled Developer ID certificate to sign it with, this will allow you to load your kext while debugging.
kcsuffix=development allows us to boot with the development kernel that we previously copied on the system (change to kcsuffix=debug to boot the debug one).
pmuflags=1 disables the watchdog timer. The Kernel Programming Guide has much more info about this option.
-v enables the kernel verbose mode that will be useful when debugging.

Retrieve the VM network configuration info (Optional)

In order to connect the debugger to the VM, we will need some info about its network configuration. We will need to change a few things on the host system but on the VM we only need to retrieve the network interface parameters and note them somewhere.

damien$ ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
  options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
  ether 00:0d:29:56:8a:70
  inet6 fe80::20c:29ff:fe56:8a70%en0 prefixlen 64 scopeid 0x4
  inet 192.168.156.140 netmask 0xffffff00 broadcast 192.168.156.255
  nd6 options=1<PERFORMNUD>
  media: autoselect (1000baseT <full-duplex>)
  status: active
...

Make sure to write down the MAC address (ether, here 00:0d:29:56:8a:70) and IP address (inet, here 192.168.156.140) somewhere.

Invalidate the kext cache

We need to invalidate the kext cache to use the new kernel for debugging. This can easily be done by running the following command:

# / means the root of the current volume, you can specify another volume if needed
damien$ sudo kextcache -invalidate /

Reboot

Finally, we need to reboot the VM for all these changes to get picked up. If your system is then stuck on some console output, feel free to shut it down for now. We’ll look into this once we have the host setup for debugging.

damien$ sudo reboot

On the host machine system

With the VM setup, we now need to set things up for debugging on the host system.

Install Xcode

In order to debug you will need Xcode to be installed on your system. You can download it from the Apple Developer page. Make sure that you launch it at least once to accept the license and install the additional tools.

Install the Kernel Debug Kit

As we previously said, the Kernel Debug Kit will also have to be installed on the host system for the debugger to use as the target and correctly symbolicate. The installation instructions are identical as the ones for the VM above.

Update the address translation table (Optional)

Optionally, we need to add an entry to the Internet-to-Ethernet address translation table for the VM network interface. We need this for the debugger to pick up the right interface when connecting to the VM IP address. The address translation table is basically a lookup table for IP addresses to MAC addresses.

With the IP address and MAC address for the VM that we wrote down earlier, we can run the following command:

# The `-S` option will add a new entry to the table, making sure to remove any existent entry for this IP first
sudo arp -S 192.168.156.140 00:0d:29:56:8a:70

Note that we can skip this by using the DB_ARP debug boot-args option as previously explained. This option is defined in the Kernel Programming Guide as:

Allow debugger to ARP and route (allows debugging across routers and removes the need for a permanent ARP entry, but is a potential security hole)—not available in all kernels.

This essentially means that the debugger itself can take care of this for us. If you’re using 0x145 for the debug boot-args option as shown above, you won’t need to do update the table entry. If you’re worried about security, feel free to use 0x105 as the option and add the entry to the table manually.

That’s it! Everything is setup, let’s try things in practice.

In practice

In order to test our setup, let’s say that we’re debugging the kernel and want to check what is calling the hfs_vnop_setxattr function and what the arguments are (this function writes the extended attribute data for a given vnode to the HFS+ store). We will want the debugger to break whenever this function is hit.

First, we’ll have to launch LLDB on the host machine and set the target to the (local) development kernel binary located in the KDK:

damien$ lldb
(lldb) target create /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).

Next, we need to launch the VM and wait for the boot screen to appear. After a few seconds, you should see the following message printed to the screen:

Darwin Bootstrapper Version 2.0.2: Sun Jul 5 21:53:13 PDT 2015; root:libxpc_executables-559.40.1~1/launchd/RELEASE_X86_64
boot-args = debug=0x001 kext-dev-mode=1 kcsuffix=development -v
IOKernelDebugger: registering debugger

ethernet MAC address: 00:0d:29:56:8a:70
ip address: 192.168.156.140

Waiting for remote debugger connection.

We can confirm that the MAC and IP addresses of the VM are the one that we noted before.

At this point, the host VM system is basically waiting for a debugger connection. Back to the LLDB session on the host machine, we can connect to the VM kernel by running the following command:

(lldb) kdp-remote 192.168.156.140

The VM will now print Connected to remote debugger and some info will also be printed to the LLDB console output.

(lldb) kdp-remote 192.168.156.140
Version: Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/DEVELOPMENT_X86_64; UUID=C75BDFDD-9F27-3694-BB80-73CF991C13D8; stext=0xffffff8011200000
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff8011200000
Kernel slid 0x11000000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 50 kext modules .................................................. done.
Target arch: x86_64
Instantiating threads completely from saved state in memory.
Process 1 stopped
* thread #2: tid = 0x0123, 0xffffff80112bb4c8 kernel.development`kdp_register_send_receive(send=<unavailable>, receive=<unavailable>) + 392 at kdp_udp.c:463, name = '0xffffff801eae94d0', queue = '0x0', stop reason = signal SIGSTOP
    frame #0: 0xffffff80112bb4c8 kernel.development`kdp_register_send_receive(send=<unavailable>, receive=<unavailable>) + 392 at kdp_udp.c:463

At this point, we are stopped in the debugger and the VM kernel is waiting for us to continue. We will first set up our breakpoint:

(lldb) breakpoint set --name hfs_vnop_setxattr
Breakpoint 1: where = kernel.development`hfs_vnop_setxattr + 34 at hfs_xattr.c:711, address = 0xffffff800b548da2

We can now continue for the kernel to complete booting:

(lldb) continue
Process 1 resuming

A bunch of text will be output on both the debugger and the system console and eventually the VM system will boot. Shortly after booting (or likely during the boot process), our breakpoint should be hit.

Process 1 stopped
* thread #15: tid = 0x0195, 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711, name = '0xffffff802177d1b0', queue = '0x0', stop reason = breakpoint 1.1
    frame #0: 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711

Back in the LLDB debugger we can print information about the current stack trace and arguments:

(lldb) thread backtrace
* thread #15: tid = 0x0195, 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711, name = '0xffffff802177d1b0', queue = '0x0', stop reason = breakpoint 1.1
  * frame #0: 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711
    frame #1: 0xffffff801394df64 kernel.development`VNOP_SETXATTR(vp=0xffffff8021dcc000, name=<unavailable>, uio=<unavailable>, options=<unavailable>, ctx=<unavailable>) + 84 at kpi_vfs.c:5162
    frame #2: 0xffffff801394165d kernel.development`vn_setxattr(vp=0xffffff8021dcc000, name=0xffffff7f94247d3d, uio=0xffffff80e230b780, options=8, context=0xffffff8020ff9d90) + 589 at vfs_xattr.c:216
    frame #3: 0xffffff8013d21a3d kernel.development`mac_vnop_setxattr(vp=0xffffff8021dcc000, name=<unavailable>, buf=0xffffff8022813008, len=29) + 189 at mac_vfs_subr.c:173
    frame #4: 0xffffff7f94246777 Quarantine`quarantine_set_ea + 125
    frame #5: 0xffffff7f94245f37 Quarantine`hook_vnode_notify_open + 430
    frame #6: 0xffffff8013d19645 kernel.development`mac_vnode_notify_open(ctx=<unavailable>, vp=0xffffff8021dcc000, acc_flags=35) + 181 at mac_vfs.c:405
    frame #7: 0xffffff801393ea6a kernel.development`vn_open_auth [inlined] vn_open_auth_finish(vp=<unavailable>, fmode=35, ctx=<unavailable>) + 14 at vfs_vnops.c:193
    frame #8: 0xffffff801393ea5c kernel.development`vn_open_auth(ndp=<unavailable>, fmodep=0xffffff80e230b9fc, vap=<unavailable>) + 2220 at vfs_vnops.c:614
    frame #9: 0xffffff801392a095 kernel.development`open1(ctx=<unavailable>, ndp=0xffffff80e230bbf0, uflags=<unavailable>, vap=0xffffff80e230bd88, fp_zalloc=<unavailable>, cra=<unavailable>, retval=<unavailable>) + 549 at vfs_syscalls.c:3344
    frame #10: 0xffffff801392ac70 kernel.development`open [inlined] open1at(fp_zalloc=<unavailable>, cra=<unavailable>) + 32 at vfs_syscalls.c:3541
    frame #11: 0xffffff801392ac50 kernel.development`open [inlined] openat_internal(ctx=0xffffff8020ff9d90, path=140423473571856, flags=546, mode=<unavailable>, fd=-2, segflg=UIO_USERSPACE, retval=0xffffff8020ff9ca0) + 281 at vfs_syscalls.c:3674
    frame #12: 0xffffff801392ab37 kernel.development`open [inlined] open_nocancel + 18 at vfs_syscalls.c:3689
    frame #13: 0xffffff801392ab25 kernel.development`open(p=<unavailable>, uap=<unavailable>, retval=0xffffff8020ff9ca0) + 85 at vfs_syscalls.c:3682
    frame #14: 0xffffff8013c2c0c1 kernel.development`unix_syscall64(state=0xffffff8021009ea0) + 753 at systemcalls.c:368
    frame #15: 0xffffff801380e656 kernel.development`hndl_unix_scall64 + 22

(lldb) p *(struct vnop_setxattr_args *)$rdi
(struct vnop_setxattr_args) $71 = {
  a_desc = 0xffffff8013e61430
  a_vp = 0xffffff8021dcc000
  a_name = 0xffffff7f94247d3d "com.apple.quarantine"
  a_uio = 0xffffff80e230b780
  a_options = 8
  a_context = 0xffffff8020ff9d90
}

Almost magical!

It’s important to note that once the kernel has launched and the debugger continued, the kernel cannot be halted again from the debugger. In fact, if you try you will get an error message:

(lldb) process interrupt
error: Failed to halt process: KDP cannot interrupt a running kernel

For this reason, you should make sure that all your breakpoints are registered in the debugger before running continue for the kernel to complete its boot.

A note about Non-Maskable Interrupts (NMI)

In the section about boot-args, we discussed using (DB_HALT | DB_ARP | DB_LOG_PI_SCRN) as the flag for the debug option. This flag contains DB_HALT that gets the kernel to wait for a debugger to attach when it boots.

As you can imagine, it is not always necessary to attach a debugger at system boot and often one only wants to attach at panic time (or even at execution time). Luckily, there’s an option to drop into the debugger on an NMI: DB_NMI.

In order to do this, we will have to change the flags for the debug boot-args option to 0x144 which is (DB_NMI | DB_ARP | DB_LOG_PI_SCRN).

With this in place, the VM kernel will boot normally without waiting for a debugger to attach. Then, at any time during execution you can generate a NMI which gives an opportunity for the remote debugger to connect.

The way to generate a NMI from a virtual machine is not obvious. Older versions of OS X would use the power button only, as documented in the Technical QA1264. However, as discussed in the WWDC 2013 session 707 this was changed to Left Command + Right Command + Power on OS X 10.9. I don’t think it is possible to simulate the power button being pressed on VMware Fusion so, at first, I thought it was actually impossible to generate a NMI from the virtual machine.

However, after reading through the Kernel Programming Guide in more details I noticed that the DB_NMI option was documented as:

Drop into debugger on NMI (Command–Power, Command-Option-Control-Shift-Escape, or interrupt switch).

And indeed, using Command-Option-Control-Shift-Escape works beautifully in the VM!

If you hit this key combo in your VM at any time (including at panic time), execution will stop. Back to the host system, you can fire up LLDB and connect to the VM.

(lldb) target create /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).

(lldb) kdp-remote 192.168.156.140
Version: Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/DEVELOPMENT_X86_64; UUID=C75BDFDD-9F27-3694-BB80-73CF991C13D8; stext=0xffffff8008800000
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff8008800000
Kernel slid 0x8600000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 93 kext modules ............................................................................................ done.
Target arch: x86_64
Instantiating threads completely from saved state in memory.
Process 1 stopped
* thread #2: tid = 0x00b8, 0xffffff80089f4e77 kernel.development`Debugger(message=<unavailable>) + 759 at model_dep.c:1013, name = '0xffffff8015e169a0', queue = '0x0', stop reason = signal SIGSTOP
    frame #0: 0xffffff80089f4e77 kernel.development`Debugger(message=<unavailable>) + 759 at model_dep.c:1013

And that’s it, as you can see that a SIGSTOP was sent in the VM kernel and back in the host debugger that’s where we’ve attached. From here we can do anything we would do in a debugger: inspect memory, print a backtrace, set a few breakpoints, etc… Once you’re done, you simply have to continue for the kernel to resume executing.

A note about kernel extensions and kernel patches

In this example, we’ve only set up breakpoints and inspected memory of the release kernel. If you plan to run a modified build of the kernel or more importantly if you are building a kernel extension, you will have to copy the .dSYM symbol file built along your kext to the host machine. It can be located anywhere that is indexed by Spotlight. When encountering an unknown symbol (for example a function in your kext), LLDB will look for a .dSYM file that matches this symbol’s Mach-O binary UUID. If the .dSYM file is on your host machine and was indexed by Spotlight then LLDB will symbolicate things nicely.

Conclusion

In this article, we presented how to set up an environment for kernel debugging. While it might sound daunting at first, we saw that it is actually pretty trivial to set up and once everything is in place, debugging the kernel is not more difficult than debugging a simple user-space application.

Setup

On the virtual machine

Install OS X 10.10

Install the Kernel Debug Kit

Install the development kernel

Update the NVRAM boot-args

Retrieve the VM network configuration info (Optional)

Invalidate the kext cache

Reboot

On the host machine system

Install Xcode

Install the Kernel Debug Kit

Update the address translation table (Optional)

In practice

A note about Non-Maskable Interrupts (NMI)

A note about kernel extensions and kernel patches

Conclusion

Update the NVRAM `boot-args`