Python
Performance

Finding Python Performance Bottlenecks

Profiling with cProfile

"cProfile (opens in a new tab) and profile provide deterministic profiling of Python programs. A profile is a set of statistics that describes how often and for how long various parts of the program executed. These statistics can be formatted into reports via the pstats (opens in a new tab) module."

Simple Approach

"""
Create a bottleneck
"""
import time
 
 
def initial_setup():
    a = 7
    time.sleep(1)
    return a
 
 
def slow_function():
    x = range(5)
    for i in x:
        a = initial_setup()
        b = a + i
        print(b)
 
 
"""
Use the simple `runcall` method that takes a function as an argument.
"""
 
import cProfile
 
profile = cProfile.Profile()
profile.runcall(slow_function)
 
"""
Pass the `profile` instance to the constructor of `Stats` to create a new instance of "ps".
We can then print the output to stdout.
"""
 
import pstats
 
ps = pstats.Stats(profile)
ps.print_stats()

Understanding the pstats output

   Random listing order was used
 
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        5    0.000    0.000    5.024    1.005 scratch_1.py:7(initial_setup)
        1    0.000    0.000    5.024    5.024 scratch_1.py:13(slow_function)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        5    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        5    5.024    1.005    5.024    1.005 {built-in method time.sleep}
NameDescription
ncallsThe number of times the analysed function has been called
tottimeThe total execution time spent in the analysed function (excluding the execution time of the subfunctions)
percallTottime divided by ncalls
cumtimeThe total execution time spent in the analysed function (including the execution time of the subfunctions)
percallcumtime dived by ncalls
filename:lineno(function)file, line number and analysed function

Profiling specific blocks of code

Using a context manager, the scope of the "profile" instance can be enclosed in a block

import cProfile
import pstats
import time
 
 
def initial_setup():
    a = 7
    time.sleep(1)
    return a
 
 
def slow_function():
    x = range(5)
    for i in x:
        a = initial_setup()
        b = a + i
        print(b)
 
 
with cProfile.Profile() as profile:
    x = range(5)
    for i in x:
        a = initial_setup()
        b = a + i
        print(b)
    ps = pstats.Stats(profile)
    ps.sort_stats("tottime", "cumtime")  # Sort by fields
    ps.print_stats(3)  # Limit output to top 3
Secret KeysBinary Search