Python Global Interpreter Lock

Disclaimer: This note is entirely based on the blog post above.
What is GIL?
The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter.
Why do we need it in Python?
Python uses reference counting for memory management. It means that objects created in Python have a reference count variable that keeps track of the number of references that point to the object. When this count reaches zero, the memory occupied by the object is released.
1
>>> import sys
2
>>> a = []
3
>>> b = a
4
>>> sys.getrefcount(a)
5
3
Copied!
The problem is that this reference count variable needed protection from race conditions where two threads increase or decrease its value simultaneously. If this happens, it can cause either leaked memory that is never released or, even worse, incorrectly release the memory while a reference to that object still exists. This can can cause crashes or other “weird” bugs in your Python programs.
This reference count variable can be kept safe by adding locks to all data structures that are shared across threads so that they are not modified inconsistently.
The impact on multi-threaded Python programs
1
# single_threaded.py
2
import time
3
from threading import Thread
4
​
5
COUNT = 100000000
6
​
7
def countdown(n):
8
    while n>0:
9
        n -= 1
10
​
11
start = time.time()
12
countdown(COUNT)
13
end = time.time()
14
​
15
print('Time taken in seconds -', end - start) # Took 6.608175992965698 on my Mac
Copied!
Now if we modified the code a bit to do to the same countdown using two threads in parallel: 
1
# multi_threaded.py
2
import time
3
from threading import Thread
4
​
5
COUNT = 100000000
6
​
7
def countdown(n):
8
    while n>0:
9
        n -= 1
10
​
11
t1 = Thread(target=countdown, args=(COUNT//2,))
12
t2 = Thread(target=countdown, args=(COUNT//2,))
13
​
14
start = time.time()
15
t1.start()
16
t2.start()
17
t1.join()
18
t2.join()
19
end = time.time()
20
​
21
print('Time taken in seconds -', end - start) # Took 6.042619943618774 on my Mac
Copied!
As you can see, both versions take almost same amount of time to finish. In the multi-threaded version the GIL prevented the CPU-bound threads from executing in parallel.
How to deal with Python’s GIL
The most popular way is to use a multi-processing approach where you use multiple processes instead of threads. Each Python process gets its own Python interpreter and memory space so the GIL won’t be a problem.
1
from multiprocessing import Pool
2
import time
3
​
4
COUNT = 50000000
5
def countdown(n):
6
    while n>0:
7
        n -= 1
8
​
9
if __name__ == '__main__':
10
    pool = Pool(processes=2)
11
    start = time.time()
12
    r1 = pool.apply_async(countdown, [COUNT//2])
13
    r2 = pool.apply_async(countdown, [COUNT//2])
14
    pool.close()
15
    pool.join()
16
    end = time.time()
17
    print('Time taken in seconds -', end - start) # Took 4.294095039367676
Copied!
We can observe a decent performance increase compared to the multi-threaded version.

The Actor Model

About Research and PhD

Last modified 2yr ago

Copy link

Contents

What is GIL?

Why do we need it in Python?

The impact on multi-threaded Python programs

How to deal with Python’s GIL