Advanced python CoP¶
Table of contents:¶
- Tools
- General tips
- Comprehensions vs filter/map
- String formatting
- All asterisk usecase
- Everything is an object
- Duck typing
- Monkey patching
- Decorators
- Memoization
Python is a religion almost as popular as Islam¶
In [1]:
%%HTML
<blockquote class="reddit-card" data-card-created="1570178057"><a href="https://www.reddit.com/r/ProgrammerHumor/comments/cqhqwa/oh_my_using_system/">Oh my using System;</a> from <a href="http://www.reddit.com/r/ProgrammerHumor">r/ProgrammerHumor</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>
Python is super easy¶
In [2]:
%%HTML
<blockquote class="reddit-card" data-card-created="1570177684"><a href="https://www.reddit.com/r/ProgrammerHumor/comments/6ee70n/whats_the_point_in_declaring_the_data_type_of_a/">What's the point in declaring the data type of a variable anyway?</a> from <a href="http://www.reddit.com/r/ProgrammerHumor">r/ProgrammerHumor</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>
...and is quite fast, because using C libraries from python is super easy¶
In [3]:
%%HTML
<blockquote class="reddit-card" data-card-created="1570177858"><a href="https://www.reddit.com/r/ProgrammerHumor/comments/dbuo2k/its_share_time/">It's share time</a> from <a href="http://www.reddit.com/r/ProgrammerHumor">r/ProgrammerHumor</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>
Tools used during presentation:¶
- ipython (enhanced interactive python shell)
- Jupyter (ipython + multimedia in the browser)
- black (uncompromising python formatter)
- isort (import sorting)
In [11]:
! pip3.8 install --user black blackcellmagic isort
In [8]:
%load_ext blackcellmagic
In [9]:
# in:
import os
def very_important_function(template: str, *variables, file: os.PathLike, engine: str, header: bool = True, debug: bool = False):
"""Applies `variables` to the `template` and writes to `file`."""
with open(file, 'w') as f:
...
In [ ]:
%%black
# out:
def very_important_function(
template: str,
*variables,
file: os.PathLike,
engine: str,
header: bool = True,
debug: bool = False,
):
"""Applies `variables` to the `template` and writes to `file`."""
with open(file, "w") as f:
...
Live pycharm project setup¶
How to choose pip package¶
- Popularity indexes:
- Check if project is alive on github - stars / last commit date
When docs isn’t enough (or there isn’t any) and there’s no stackoverflow question :(¶
Example function usages from open source projects: https://www.programcreek.com/python/
Recomended python blog: https://realpython.com
In [12]:
%%HTML
<img src="https://i.imgflip.com/3ca7td.jpg" title="made at imgflip.com"/>
Python compiles source code .py files to python bytecode .pyc files (__pycache__ directory). Bytes code is then interpreted by python.
The zen of python¶
In [6]:
import this
In [7]:
numbers = [1, 2, 3, 4]
In [8]:
def is_even(n):
return n % 2 == 0
In [9]:
list(filter(is_even, numbers))
Out[9]:
In [10]:
list(map(is_even, numbers))
Out[10]:
In [11]:
[number * 2 for number in range(10) if number % 2 != 0]
Out[11]:
In [12]:
list(map(lambda number: number * 2, filter(lambda number: number % 2 != 0, range(10))))
Out[12]:
Btw, there's also dict comprehension
In [14]:
list(zip(names, surnames))
Out[14]:
In [13]:
names = ["Janusz", "Guido"]
surnames = ["Pawlak", "van Rossum"]
{name: surname for name, surname in zip(names, surnames)}
Out[13]:
Python oneliners are great
In [15]:
import os
open("key", "wb").write(os.urandom(16))
Out[15]:
Read binary file and output it in uppercase hex strnig with spaces in between:
In [16]:
" ".join(f"{byte_:02X}" for byte_ in open("key", "rb").read())
Out[16]:
In [17]:
%%HTML
<blockquote class="reddit-card" data-card-created="1570177536"><a href="https://www.reddit.com/r/ProgrammerHumor/comments/d46pmg/from_hell_import_depression_as_magic/">from hell import depression as magic</a> from <a href="http://www.reddit.com/r/ProgrammerHumor">r/ProgrammerHumor</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>
String formatting¶
In [18]:
x = 10
In [19]:
"x = %d" % x # python2 style
Out[19]:
In [20]:
"x = " + str(x) # get's complex with multiple concatenation
Out[20]:
In [21]:
"x = {}".format(x)
Out[21]:
In [22]:
"x = {x}".format(x=x)
Out[22]:
In [23]:
f"x = {x}" # python 3.6
Out[23]:
In [24]:
f"{x + 2 * 2}" # any valid expression works
Out[24]:
In [25]:
"x = {x}".format(**vars())
Out[25]:
In [26]:
f"{x=}" # python 3.8
Out[26]:
In [27]:
f"{x + 2 * 2 =}" # python 3.8
Out[27]:
In [28]:
2 * 3
Out[28]:
In [29]:
2 ** 3
Out[29]:
Variable number of arguments (*args, **kwargs)¶
In [30]:
def f(*args):
print(f"{type(args) = }")
print(f"{args = }")
f(1, "a", [1,2,3])
In [31]:
def f(x, *just_an_identifier):
print(f"{type(just_an_identifier) = }")
print(f"{x = }")
print(f"{just_an_identifier = }")
f(1, "a", [1, 2, 3])
In [13]:
def f(x, *args, **kwargs):
print(f"{type(kwargs) = }")
print(f"{locals() = }")
f(1, "a", [1, 2, 3], named_param="b")
In [14]:
f(1, "a", named_param="b", [1, 2, 3])
Unpacking in function calls¶
Problem:
In [37]:
point = (1, 2)
def f(x, y):
print(f"{locals() = }")
f(point[0], point[1])
In [38]:
f(*point)
In [39]:
f(*"ab")
Same goes for dict:
In [16]:
def f(a, b, c):
print(f"{locals() = }")
kwargs = {"a": 1, "b": 2, "c": 3}
# you can also generate the same dict using zip :D
kwargs = dict(zip("abc", range(1, 4)))
f(a = kwargs['a'], b = kwargs['b'], c = kwargs['c'])
In [41]:
f(**kwargs)
Most common usecase - parameter forwarding:
In [42]:
def forward_params(func, *args, **kwargs):
func(*args, **kwargs)
Using the same argument set in many function calls:
In [17]:
import subprocess
subprocess.run("ls", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
subprocess.run("whoami", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
subprocess.run("pwd", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, check=True)
Out[17]:
In [18]:
run_kwargs = {"stdout": subprocess.PIPE, "stderr": subprocess.PIPE, "shell": True}
subprocess.run("ls", **run_kwargs)
subprocess.run("whoami", **run_kwargs)
subprocess.run("pwd", **run_kwargs, check=True)
Out[18]:
Extended unpacking¶
In [18]:
point = (1, 2, 3)
x, y, z = point
for var_name in "xyz":
print(f"{var_name} = {locals()[var_name]}")
In [46]:
point3d = (1, 2, 3)
*point2d, z = point3d
print(f"{z = }")
print(f"{point2d = }")
In [19]:
student_with_grades = ("Antoni", 2, 2, 3, 2, 2)
student, *grades = student_with_grades
print(f"{student = }")
print(f"{grades = }")
In [21]:
student_with_grades = ("Antoni", 2, 2, 3, 2, 2)
_, *grades = student_with_grades
print(f"{_ = }")
print(f"{grades = }")
_ is just an identifier. The same as grades. It's a common practice to use _ when you want to ignore some value
Named only arguments¶
In [22]:
def f(*, a, b):
pass
f(1, 2)
In [23]:
f(a=1, b=2)
In [24]:
def f(a, *, b):
pass
f(1, 2)
In [26]:
f(1, b=2)
Btw - positional only arguments:
In [29]:
def f(a, / , b): # python 3.8
pass
f(a=1, b=2)
Btw valrus operator in python 3.8:
In [32]:
a = [0]*11
if (n := len(a)) > 10:
print(f"List is too long ({n} elements, expected <= 10)")
Everything is an object in Python¶
Functions are objects
In [55]:
def f():
if not hasattr(f, "call_count"):
f.call_count = 0
f.call_count += 1
f()
f()
print(f.call_count)
In [56]:
def Person(name):
def new_person():
pass
new_person.say_hello = lambda: f"My names is {new_person.name}"
new_person.name = name
return new_person
In [57]:
janusz = Person(name="Janusz")
In [58]:
janusz.say_hello()
Out[58]:
In [59]:
janusz.name
Out[59]:
Numbers are also objects:
In [60]:
x = 1
In [61]:
dir(x)
Out[61]:
In [62]:
from inspect import getfullargspec
getfullargspec(x.to_bytes)
Out[62]:
Function documentation shortcut in ipython
In [33]:
x.to_bytes?
In [35]:
x.to_bytes??
In [36]:
help(x.to_bytes)
In [64]:
x.to_bytes(8,'big')
Out[64]:
Everything is also an reference¶
Identity vs equality
In [66]:
a = [1, 2, 3]
b = [1, 2, 3]
In [67]:
a == b
Out[67]:
In [68]:
a is b
Out[68]:
In [69]:
x = 1
y = 1
In [70]:
x is y
Out[70]:
In [71]:
x = -25
y = -25
In [72]:
x is y
Out[72]:
That's because python preallocates most commonly used numbers (-5; 255) as singletons to improve performance.
The is keyword is mostly used to compare True, False and None objects
In [73]:
True is not None
Out[73]:
The implementation of "is" is to compare id of the compared objects. Id in Cpython is just an memory address.
In [37]:
id(None)
Out[37]:
Dynamic Duck typing¶
A reference (variable) can change the type it's pointing to:
In [40]:
x = "a"
x = 1
You can even change class of some existing object:
In [21]:
class A:
x = 10
def f(self):
return "A"
class B:
def f(self):
return "B"
a, b = A(), B()
In [22]:
a.__class__ = b.__class__
In [23]:
a.f()
Out[23]:
In [24]:
a.x
In [25]:
a.x = 10
In [26]:
a.x
Out[26]:
Monkey patching¶
In [27]:
a.f = lambda: "A"
a.f()
Out[27]:
In [31]:
a.f = B.f
In [32]:
a.f(a)
Out[32]:
Python is a really dynamic language, almost everything can change in runtime - so no optimizations¶
Python has to compute the same value in a loop, because you could change the function from different thread:
In [85]:
from threading import Timer
from time import sleep
def some_long_function(a, b):
sleep(0.3)
return a + b
def break_long_function():
global some_long_function
some_long_function = None
Timer(0.6, break_long_function).start()
for i in range(10000000):
print(i)
some_long_function(1, 1)
Decorators¶
Useful when you want to do something before or after each function call. For example, measuring execution time:
In [33]:
from datetime import datetime
def call_and_print_time(func):
now = datetime.now()
func()
print("Execution time: ", datetime.now() - now)
def f():
print("Doing work")
from time import sleep
sleep(0.3)
call_and_print_time(f)
In [86]:
def decorator(f):
return f
@decorator
def decorated():
pass
It works because @ is just a syntactic sugar for:
In [87]:
decorated = decorator(decorated)
Simplest functional decorator
In [88]:
from datetime import datetime
def call_and_print_time(func):
now = datetime.now()
func()
print("Execution time: ", datetime.now() - now)
def execution_time_printed(func):
return lambda: call_and_print_time(func)
@execution_time_printed
def f():
print("Doing work")
from time import sleep
sleep(0.3)
f()
But since functions can be defined inside other functions it's a common practice to do it this way:
In [89]:
from datetime import datetime
def execution_time_printed(func):
def call_and_print_time():
now = datetime.now()
func()
print("Execution time: ", datetime.now() - now)
return call_and_print_time
@execution_time_printed
def f():
print("Doing work")
from time import sleep
sleep(0.3)
f()
Let's add arguments delegation and return value
In [90]:
from datetime import datetime
def execution_time_printed(func):
def call_and_print_time(*args, **kwargs):
now = datetime.now()
return_value = func(*args, **kwargs)
print("Execution time: ", datetime.now() - now)
return return_value
return call_and_print_time
@execution_time_printed
def add(x, y):
return x + y
add(1, 2)
Out[90]:
In [91]:
add.__name__
Out[91]:
In [92]:
"\U0001f622"
Out[92]:
We could fix this by adding call_and_print_time.__name__ = func.__name__ but there's more metadata to fix (i.e. docstring). @wraps to the rescue
In [93]:
from datetime import datetime
from functools import wraps
def execution_time_printed(func):
@wraps(func)
def call_and_print_time(*args, **kwargs):
now = datetime.now()
return_value = func(*args, **kwargs)
print("Execution time: ", datetime.now() - now)
return return_value
return call_and_print_time
@execution_time_printed
def add(x, y):
return x + y
add.__name__
Out[93]:
Here comes the real inception - decorator with arguments have to return a decorator function
@execution_time_printed(file=sys.stderr) <- bolded must return a decorator
In [42]:
def decorator():
def real_decorator(func):
return func
return real_decorator
@decorator()
def decorated():
pass
decorated = decorator()(decorated)
In [43]:
import io
import sys
from datetime import datetime
from functools import wraps
def execution_time_printed(file=sys.stdout):
def decorator(func):
@wraps(func)
def call_and_print_time(*args, **kwargs):
now = datetime.now()
return_value = func(*args, **kwargs)
print("Execution time: ", datetime.now() - now, file=file)
return return_value
return call_and_print_time
return decorator
string_stream = io.StringIO()
@execution_time_printed(file=string_stream)
def add(x, y):
return x + y
add(1, 2)
string_stream.seek(0)
print(f"{string_stream.read() = }")
But now you always needs to call the decorator
In [44]:
@execution_time_printed
def add(x, y):
return x + y
add(1,2)
In [45]:
@execution_time_printed()
def add(x, y):
return x + y
add(1,2)
Out[45]:
That's how all decorators with arguments should be implemented:
In [46]:
import io
import sys
from datetime import datetime
from functools import wraps
def execution_time_printed(func_=None, file=sys.stdout):
def decorator(func):
@wraps(func)
def call_and_print_time(*args, **kwargs):
now = datetime.now()
return_value = func(*args, **kwargs)
print("Execution time: ", datetime.now() - now, file=file)
return return_value
return call_and_print_time
if func_ is None:
return decorator
else:
return decorator(func_)
@execution_time_printed
def add(x, y):
return x + y
add(1, 2)
Out[46]:
In [47]:
@execution_time_printed()
def add(x, y):
return x + y
add(1, 2)
Out[47]:
In [48]:
@execution_time_printed(file=sys.stderr)
def add(x, y):
return x + y
add(1, 2)
Out[48]:
Create an object with a variable that specifies the type:
In [49]:
cls = list
In [50]:
list([1,2,3])
Out[50]:
In [51]:
cls([1,2,3])
Out[51]:
Classes can also be decorated:
In [52]:
import functools
def singleton(cls):
"""Make a class a Singleton class (only one instance)"""
@functools.wraps(cls)
def wrapper_singleton(*args, **kwargs):
if not wrapper_singleton.instance:
wrapper_singleton.instance = cls(*args, **kwargs)
return wrapper_singleton.instance
wrapper_singleton.instance = None
return wrapper_singleton
@singleton
class TheOne:
pass
a, b = TheOne(), TheOne()
a is b
Out[52]:
In [35]:
from itertools import count
def add_id(original_class):
orig_init = original_class.__init__
id_counter = count()
def __init__(self, *args, **kws):
self.id = next(id_counter)
orig_init(self, *args, **kws)
original_class.__init__ = __init__
return original_class
@add_id
class Person:
def __init__(self, name):
self.name = name
In [36]:
Person('Janusz').__dict__
Out[36]:
In [37]:
vars(Person('Guido'))
Out[37]:
Example of class decorator from standard library - dataclass.
In [108]:
from dataclasses import dataclass, field
@dataclass()
class Person:
name: str
surname: str
id = field
In [55]:
import functools
class CountCalls:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
self.num_calls = 0
def __call__(self, *args, **kwargs):
self.num_calls += 1
print(f"Call {self.num_calls} of {self.func.__name__!r}")
return self.func(*args, **kwargs)
@CountCalls
def say_whee():
print("Whee!")
say_whee()
say_whee()
Inheritance with decorator :D (Don't do this)
In [56]:
def inherit_from(*bases):
def wrapper(cls):
class Derived(cls, *bases):
pass
return Derived
return wrapper
class A:
def f(self):
print("Hello world")
@inherit_from(A)
class B:
pass
B().f()
Memoization - caching returned values¶
In [60]:
def factorial(n: int)-> int:
if n < 2:
return n
return n * factorial(n-1)
In [61]:
%%timeit
factorial(1000)
In [1]:
def cache_factorial(n: int) -> int:
if n not in cache_factorial.__cache__:
cache_factorial.__cache__[n] = n * cache_factorial(n - 1)
return cache_factorial.__cache__[n]
cache_factorial.__cache__ = {1: 1}
Now the second call with the same params will return cached value:
In [2]:
%time _ = cache_factorial(1000)
In [3]:
%time _ = cache_factorial(1000)
In [63]:
%%timeit
cache_factorial(1000)
This mechanism can be generalised into a very simple decorator:
In [64]:
import functools
def cache(func):
"""Keep a cache of previous function calls"""
@functools.wraps(func)
def wrapper_cache(*args, **kwargs):
cache_key = args + tuple(kwargs.items())
if cache_key not in wrapper_cache.cache:
wrapper_cache.cache[cache_key] = func(*args, **kwargs)
return wrapper_cache.cache[cache_key]
wrapper_cache.cache = dict()
return wrapper_cache
@cache
def factorial(n):
if n < 2:
return n
return n * factorial(n-1)
In [65]:
%%timeit
factorial(1000)
There is a similar decorator in python standard library:
In [4]:
from functools import lru_cache
@lru_cache(maxsize=1010)
def factorial(n):
if n < 2:
return n
return n * factorial(n - 1)
In [5]:
%%timeit
factorial(1000)
It's faster than my implementation because it's implemented in C :P
https://github.com/python/cpython/blob/master/Modules/_functoolsmodule.c#L863
Typing¶
Sometimes the expected object type can tell you more than documentation. It's also very useful to make code autocomplete in your IDE more inteligent:
In [8]:
from typing import Optional, Union, List, Dict, Any
def f(a: int, b: List[int], c: Dict[str, List[str]], e: Any, d: Optional[str] = None) -> Union[int, str]:
if a == 1:
return 1
return "1"
In [9]:
def get_int():
return 2
In [10]:
x : int = get_int()
typechecked decorator - how to make python strong typed language (only in runtime :( )¶
In [11]:
import functools
from inspect import getfullargspec
def typechecked(func):
@functools.wraps(func)
def wrapper(**kwargs):
argspec = getfullargspec(func)
for arg_name, declared_type in argspec.annotations.items():
if not isinstance(kwargs.get(arg_name), declared_type):
raise RuntimeError(
f"error: invalid conversion from {type(kwargs.get(arg_name))} to {declared_type} [-fpermissive]"
)
return func(**kwargs)
return wrapper
@typechecked
def f(a: int, b: str):
pass
In [12]:
f(a=1, b="2")
In [13]:
f(a="2", b=1)
It's implemented in: https://github.com/prechelt/typecheck-decorator
Make the whole module type-checked :D
In [14]:
import types
import functools
def decorate_all_in_module(module, decorator):
for name in dir(module):
obj = getattr(module, name)
if isinstance(obj, types.FunctionType):
setattr(module, name, decorator(obj))
import some_module
decorate_all_in_module(some_module, typechecked)
In [15]:
some_module.f1(int_arg="a")
In [16]:
%%HTML
<blockquote class="reddit-card" data-card-created="1570178753"><a href="https://www.reddit.com/r/ProgrammerHumor/comments/d900nx/ah_yespython_finally_pays_out/">ah yes...python finally pays out.</a> from <a href="http://www.reddit.com/r/ProgrammerHumor">r/ProgrammerHumor</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>