How to debug deadlocks in Python programs with Visual Studio Code

Imagine a program with two threads (T1, T2), each working on tasks that require exclusive access to two resources (R1, R2). If T1 acquires R1 and then tries to get R2, it is possible that at the same time, T2 has already gotten hold of R2. T2 can not progress because it needs R1, currently owned by T1. In this situation, they will wait for each other forever or until they are terminated by a higher authority.

Most deadlock situations in software systems are more complex than this one, but the principle is the same: two or more parties cannot progress because the management of shared resources is incorrect. We will not cover resource allocation, locking, and synchronization algorithms here, but it is important to remember that synchronization objects like locks, semaphores, etc. themselves are shared resources.

Synchronization problems are often difficult to analyze or spot in automated or manual tests. They are notoriously hard, often hidden, and then occur sporadically. Therefore, they can surface late, maybe a long time after the software has been shipped to customers. That makes it necessary to analyze them whenever they are observed in real life.

Example

We will create a deadlock situation. Consider this Python function:

def worker(colleague_working):
    while colleague_working:
        sleep(1)

It simulates a team member who is hanging around, and appears to be busy but does nothing except check every second if someone else is at work. It makes no sense to stay at work if nobody notices it. He will leave as soon as he is alone. The function parameter colleague_working is being passed to the function by reference so that it can change at any time. It will take the value False when nobody else is working.

To make matters worse, we will employ two lazy workers. We simulate this by running them in parallel threads. Each one is aware of the other’s presence. This will create a deadlock situation because the two are waiting for each other to go home.

Creating a new thread in Python happens in two steps. First, an object of type ‘Thread’ must be created. This object gets a callable (function-like) object that it executes and, optionally, any parameters to this callable. In our simulation, there are two parameters to the thread function, references to our own work status and to that of the colleague. The first one could be modified, while the second will only be read. A thread is started by calling the Thread.run() method. It will run until the thread function terminates.

Now we let two threads communicate via two shared variables, busy_1 and busy_2:

busy_1 = True
busy_2 = True
worker_1 = Thread(target=worker, args=[busy_2, busy_1])
worker_2 = Thread(target=worker, args=[busy_1, busy_2])
worker_1.start()
worker_2.start()

The main program is a third thread of execution. It is responsible for creating and managing the workers. It will terminate after the two workers have terminated. The entire program:

In this example, the main program (or GUI) freezes; an unpleasant user experience.

fig. 1: Two threads waiting for each other

Using the Debugger

We are lucky if our program is ‘freezing’ in a debugger session. We can just hit the pause button and analyze the situation. Here we are using the Visual Studio Code editor with the Python plugin.

fig. 2: A Python process with three threads

In the debugger, we can see a list of threads. T1 and T2 are both trapped in a loop until colleague_working takes the value False. We have given names to the threads to easily identify them. In the two call stacks, we will also see the value of the function parameter. We can select threads within a paused process. Each thread has its own call stack.

fig. 3: Thread 1 is selected, call stack and local variables are displayed

If the program has not been launched by a debugger, we have to attach a debugger later to the running process. VS Code can handle a set of debug configurations. Their properties are stored in the .vscode/launch.json.

In this file, we have two configurations. The first one is of type “launch”. It will start the current python file in the debugger. The second one is of type “attach”. It will open a list of running processes when selected and run. Our Python program will be the first argument to a running “Python.exe” instance in this list. Now we have to find our process, select it, and hit the Pause ‘||’ button. We can now investigate the call stacks of its threads.

fig. 4: Attach

Originally published at http://github.com.