Skip to content

Debugging Multithreaded Applications

Debugging multi-threaded applications can be a daunting task, particularly in an interactive debugger that is designed to operate on one thread at a time. libdebug offers a few features that will help you debug multi-threaded applications more intuitively and efficiently.

Child Threads

Threads of a running process in the POSIX standard are children of the main process. They are created by system calls such as fork, clone and clone3. In the Linux kernel, the ptrace system call allows a tracer to identify new threads of the debugged process and retrieve their thread id (tid).

libdebug automatically registers new threads and exposes their state with the same API as the main Debugger object. While technically threads can be running or stopped independently, libdebug will enforce a coherent state. This means that if a thread is stopped, all other threads will be stopped as well and if a continuation command is issued, all threads will be resumed.

To access the threads of a process, you can use the threads attribute of the Debugger object. This attribute will return a list of ThreadContext objects, each representing a thread of the process. Similarly, you can access the Debugger object from any ThreadContext through the debugger attribute.

Meaning of the debugger object

When accessing state fields of the Debugger object (e.g. registers, memory), the debugger will act as an alias for the main thread. For example, doing d.regs.rax will be equivalent to doing d.threads[0].regs.rax.

Child Processes

libdebug does not support debugging child processes (only threads). If a child process is created by the main process, a warning will be printed, prompting the user to attach to the child process manually.

Shared and Unshared State

Each thread has its own register set, stack, and instruction pointer. However, there are shared resources between threads that you should be aware of:

  • The virtual address space is mostly shared between threads. Currently, libdebug does not handle the multiprocessing.

  • Software breakpoints are implemented through code patching in the process memory. This means that a breakpoint set in one thread will be replicated across all threads.

    • When using synchronous breakpoints, you will need to "diagnose" the stopping event to determine which thread triggered the breakpoint. You can do this by checking the return value of the hit_on() method of the Breakpoint object. Passing the ThreadContext as an argument will return True if the breakpoint was hit by that thread.

    • When using asynchronous breakpoints, the breakpoint will be more intuitive to handle, as the signature of the callback function includes the ThreadContext object that triggered the breakpoint.

  • While hardware breakpoints are thread-specific, libdebug mirrors them across all threads. This is done to avoid asymmetries with software breakpoints. Watchpoints are hardware breakpoints, so this applies to them as well.

  • For consistency, syscall handlers are also enabled across all threads. The same considerations for synchronous and asynchronous breakpoints apply here as well.

Concurrency in Syscall Handling

When debugging entering and exiting events in syscalls, be mindful of the scheduling. The kernel may schedule a different thread to handle the syscall exit event right after the enter event of another thread.

  • Signal Catching is also shared among threads. Apart from consistency, this is a necessity. In fact, the kernel does not guarantee that a signal sent to a process will be dispatched to a specific thread.
    • By contrast, when sending arbitrary signals through the ThreadContext object, the signal will be sent to the requested thread.

How to access TLS?

While the virtual address space is shared between threads, each thread has its own Thread Local Storage (TLS) area. As it stands, libdebug does not provide a direct interface to the TLS area.