Python type hints from introduction to practice

Python I think everyone is familiar with it , Even arguments about whether it is useful or useless may have tired of reading . But anyway , As a language that will be added to the college entrance examination, it still has its uniqueness , Today we'll talk again Python.

Python Is a dynamic strongly typed language

《 smooth Python》 It is mentioned in the book , If a language rarely implicitly converts types , So it's a strongly typed language , for example Java、C++ and Python Is a strongly typed language .

At the same time, if a language often implicitly converts types , So it's a weakly typed language ,PHP、JavaScript and Perl It's a weakly typed language .

Of course, the above simple example comparison , Not exactly Python Is a strong type of language , because Java Also supports integer and string Add operation , And Java It's a strongly typed language . therefore 《 smooth Python》 There are also definitions of static and dynamic types ： The language for checking type at compile time is static type language , The language for checking types at run time is dynamic type language . Static languages require declaration types （ Some modern languages use type derivation to avoid partial type declarations ）.

in summary , About Python Dynamic strongly typed languages are obvious and uncontroversial .

Type Hints On

Python stay PEP 484（Python Enhancement Proposals,Python Enhancement proposal ）[www.python.org/dev/peps/pe… Type Hints（ Type notes ）. Further strengthened Python Is a feature of a strongly typed language , It's in Python3.5 It's the first time that . Use Type Hints Let's write a with types Python Code , It looks more in line with the strongly typed language style .

There are two definitions greeting function ：

• The common expression is as follows ：

name = "world"

def greeting(name):
    return "Hello " + name

greeting(name)
 Copy code 

• Joined the Type Hints Let's write it as follows ：

name: str = "world"

def greeting(name: str) -> str:
    return "Hello " + name

greeting(name)
 Copy code 

With PyCharm For example , In the process of writing code IDE It will label according to the type of function , Type check the parameters passed to the function . If you find that the argument type does not match the formal parameter type label of the function, you will be prompted as follows ：

Common data structures Type Hints How to write it

Through a greeting The function shows Type Hints Usage of , Next we will Python Common data structures Type Hints Write more in-depth study .

Default parameters

Python Function supports default arguments , The following are the default parameters Type Hints How to write it , Just write the type between the variable and the default parameter .

def greeting(name: str = "world") -> str:
    return "Hello " + name

greeting()
 Copy code 

Custom type

For custom types ,Type Hints Can also support . Its writing is similar to Python There is no difference between built-in types .

class Student(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age


def student_to_string(s: Student) -> str:
    return f"student name: {s.name}, age: {s.age}."

student_to_string(Student("Tim", 18))
 Copy code 

When the type is labeled as a custom type ,IDE You can also check the type .

Container type

When we want to add type labels to built-in container types , Because of the type annotation operator [] stay Python Represents the slice operation in , Therefore, it will cause syntax errors . So you can't use built-in container types directly as annotations , Need from typing Import the corresponding container type annotation in the module （ Usually the initial capital form of the built-in type ）.

from typing import List, Tuple, Dict

l: List[int] = [1, 2, 3]

t: Tuple[str, ...] = ("a", "b")

d: Dict[str, int] = {
    "a": 1,
    "b": 2,
}
 Copy code 

however PEP 585[www.python.org/dev/peps/pe… Python Built in type , Without syntax errors .

l: list[int] = [1, 2, 3]

t: tuple[str, ...] = ("a", "b")

d: dict[str, int] = {
    "a": 1,
    "b": 2,
}
 Copy code 

Type the alias

Some complex nested types are long to write , If there is repetition , It will be painful , The code will not be clean enough .

config: list[tuple[str, int], dict[str, str]] = [
    ("127.0.0.1", 8080),
    {
        "MYSQL_DB": "db",
        "MYSQL_USER": "user",
        "MYSQL_PASS": "pass",
        "MYSQL_HOST": "127.0.0.1",
        "MYSQL_PORT": "3306",
    },
]

def start_server(config: list[tuple[str, int], dict[str, str]]) -> None:
    ...

start_server(config)
 Copy code 

This can be solved by aliasing the type , Similar variable naming .

Config = list[tuple[str, int], dict[str, str]]


config: Config = [
    ("127.0.0.1", 8080),
    {
        "MYSQL_DB": "db",
        "MYSQL_USER": "user",
        "MYSQL_PASS": "pass",
        "MYSQL_HOST": "127.0.0.1",
        "MYSQL_PORT": "3306",
    },
]

def start_server(config: Config) -> None:
    ...

start_server(config)
 Copy code 

This makes the code look much more comfortable .

Variable parameters

Python A very flexible aspect of functions is that they support variable parameters ,Type Hints It also supports variable parameter type annotation .

def foo(*args: str, **kwargs: int) -> None:
    ...

foo("a", "b", 1, x=2, y="c")
 Copy code 

IDE Can still check out .

Generic

Using dynamic languages requires generic support ,Type Hints There are also many solutions for generics .

TypeVar

Use TypeVar Can accept any type of .

from typing import TypeVar

T = TypeVar("T")

def foo(*args: T, **kwargs: T) -> None:
    ...

foo("a", "b", 1, x=2, y="c")
 Copy code 

Union

If you don't want to use generics , Just want to use several specified types , Then you can use Union To do it . Like defining concat The function just wants to receive str or bytes type .

from typing import Union

T = Union[str, bytes]

def concat(s1: T, s2: T) -> T:
    return s1 + s2

concat("hello", "world")
concat(b"hello", b"world")
concat("hello", b"world")
concat(b"hello", "world")
 Copy code 

IDE The inspection prompt is shown in the figure below ：

TypeVar and Union difference

TypeVar Not only can you receive generics , It can also be like Union The use of , You just need to pass the type range you want to specify as a parameter in turn when instantiating . Follow Union The difference is , Use TypeVar Declared function , Multiple parameter types must be the same , and Union No restrictions .

from typing import TypeVar

T = TypeVar("T", str, bytes)

def concat(s1: T, s2: T) -> T:
    return s1 + s2

concat("hello", "world")
concat(b"hello", b"world")
concat("hello", b"world")
 Copy code 

Here are the USES TypeVar When defining types IDE Tips ：

Optional

Type Hints Provides Optional As a Union[X, None] Short form of , Indicates that the marked parameter is either X type , Either for None,Optional[X] Equivalent to Union[X, None].

from typing import Optional, Union

# None => type(None)
def foo(arg: Union[int, None] = None) -> None:
    ...


def foo(arg: Optional[int] = None) -> None:
    ...
 Copy code 

Any

Any It's a special type , Can represent all types of . Function with return value and parameter type not specified , All implicitly default to Any, So the following two greeting Function writing is equivalent ：

from typing import Any

def greeting(name):
    return "Hello " + name


def greeting(name: Any) -> Any:
    return "Hello " + name
 Copy code 

When we both want to use Type Hints To implement the writing of static types , And don't want to lose the flexibility unique to dynamic languages , You can use Any.

Any When a type value is assigned to a more precise type , Do not perform type checking , The following code IDE There will be no error message ：

from typing import Any

a: Any = None
a = []  #  Dynamic language features 
a = 2

s: str = ''
s = a  # Any  Type values are assigned to more precise types 
 Copy code 

Callable object （ function 、 Class etc. ）

Python Any callable type in can use Callable Annotate . The following code is marked Callable[[int], str],[int] Parameter list representing callable types ,str Represents the return value .

from typing import Callable

def int_to_str(i: int) -> str:
    return str(i)

def f(fn: Callable[[int], str], i: int) -> str:
    return fn(i)

f(int_to_str, 2)
 Copy code 

Quote from

When we need to define a tree structure , Self reference is often required . When executed init When the method is used Tree Type has not been generated yet , So it can't be used like str Like this built-in type, you can label directly , Need to be in string form “Tree” To reference an object that has not been generated .

class Tree(object):
    def __init__(self, left: "Tree" = None, right: "Tree" = None):
        self.left = left
        self.right = right

tree1 = Tree(Tree(), Tree())
 Copy code 

IDE Self reference types can also be checked .

This form can not only be used for self reference , Preceding references also apply .

The duck type

Python One notable feature is its extensive use of duck types ,Type Hints Provides Protocol To support duck types . When defining a class, you only need to inherit Protocol You can declare an interface type , When encountering annotation of interface type , As long as the received object implements all methods of the interface type , You can check the type annotation ,IDE There will be no error . there Stream No explicit inheritance is required Interface class , It just needs to be done close The method can .

from typing import Protocol

class Interface(Protocol):
    def close(self) -> None:
        ...

# class Stream(Interface):
class Stream:
    def close(self) -> None:
        ...

def close_resource(r: Interface) -> None:
    r.close()

f = open("a.txt")
close_resource(f)

s: Stream = Stream()
close_resource(s)
 Copy code 

Because of the built-in open Function returns the file object and Stream Objects all implement close Method , So you can pass Type Hints The inspection of , And strings “s” It didn't happen close Method , therefore IDE Type error will be prompted .

Type Hints Other ways of writing

actually Type Hints There is not only one way to write ,Python In order to be compatible with different people's preferences and the migration of old code, two other writing methods are implemented .

Write with comments

Come and have a look at tornado Examples of frameworks （tornado/web.py）. It is applicable to modification of existing projects , The code is written , Type labels need to be added later .

Write in a separate file （.pyi）

You can create a new one in the same directory as the source code .py Homonymous .pyi file ,IDE It can also automatically do type checking . The advantage of this is that you can make no changes to the original code , Completely decoupled . The disadvantage is that it is equivalent to maintaining two copies of code at the same time .

Type Hints practice

Basically , Commonly used in daily coding Type Hints The writing methods have been introduced to you , Let's take a look at how to apply it in practical coding Type Hints.

dataclass—— Data class

dataclass It's a decorator , It can decorate classes , Used to add magic methods to classes , for example init() and repr() etc. , It's in PEP 557[www.python.org/dev/peps/pe…

from dataclasses import dataclass, field


@dataclass
class User(object):
    id: int
    name: str
    friends: list[int] = field(default_factory=list)


data = {
    "id": 123,
    "name": "Tim",
}

user = User(**data)
print(user.id, user.name, user.friends)
# > 123 Tim []
 Copy code 

Above use dataclass The code written is equivalent to the following code ：

class User(object):
    def __init__(self, id: int, name: str, friends=None):
        self.id = id
        self.name = name
        self.friends = friends or []


data = {
    "id": 123,
    "name": "Tim",
}

user = User(**data)
print(user.id, user.name, user.friends)
# > 123 Tim []
 Copy code 

Be careful ：dataclass Field types are not checked .

You can find , Use dataclass Writing classes can reduce a lot of duplicate template code , Grammar is also clearer .

Pydantic

Pydantic It's based on Python Type Hints Third party library , It provides data validation 、 Serialization and documentation capabilities , It is a library worth learning from . Here's a use of Pydantic Example code for ：

from datetime import datetime
from typing import Optional

from pydantic import BaseModel


class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: Optional[datetime] = None
    friends: list[int] = []


external_data = {
    'id': '123',
    'signup_ts': '2021-09-02 17:00',
    'friends': [1, 2, '3'],
}
user = User(**external_data)

print(user.id)
# > 123
print(repr(user.signup_ts))
# > datetime.datetime(2021, 9, 2, 17, 0)
print(user.friends)
# > [1, 2, 3]
print(user.dict())
"""
{
    'id': 123,
    'signup_ts': datetime.datetime(2021, 9, 2, 17, 0),
    'friends': [1, 2, 3],
    'name': 'John Doe',
}
"""
 Copy code 

Be careful ：Pydantic The field type will be forcibly checked .

Pydantic It's written like dataclass Very similar , But it does more extra work , It also provides information such as .dict() This is a very convenient way .

Let's see another one Pydantic An example of data validation , When User When the parameter received by the class does not meet the expectation , Will throw out ValidationError abnormal , The exception object provides .json() This method is convenient to view the cause of the exception .

from pydantic import ValidationError

try:
    User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
    print(e.json())
"""
[
  {
    "loc": [
      "id"
    ],
    "msg": "field required",
    "type": "value_error.missing"
  },
  {
    "loc": [
      "signup_ts"
    ],
    "msg": "invalid datetime format",
    "type": "value_error.datetime"
  },
  {
    "loc": [
      "friends",
      2
    ],
    "msg": "value is not a valid integer",
    "type": "type_error.integer"
  }
]
"""
 Copy code 

All error messages are saved in one list in , The error report of each field is saved in the nested dict in , among loc Identifies the exception field and error reporting location ,msg It is an error message ,type Is the error reporting type , In this way, the whole reason for the error report is clear at a glance .

MySQLHandler

MySQLHandler[github.com/jianghushin… pymysql Encapsulation of Libraries , Make it support the use of with Syntax call execute Method , And the query results from tuple Replace with object, It's also true Type Hints Application .

class MySQLHandler(object):
    """MySQL handler"""

    def __init__(self):
        self.conn = pymysql.connect(
            host=DB_HOST,
            port=DB_PORT,
            user=DB_USER,
            password=DB_PASS,
            database=DB_NAME,
            charset=DB_CHARSET,
            client_flag=CLIENT.MULTI_STATEMENTS,  # execute multi sql statements
        )
        self.cursor = self.conn.cursor()

    def __del__(self):
        self.cursor.close()
        self.conn.close()

    @contextmanager
    def execute(self):
        try:
            yield self.cursor.execute
            self.conn.commit()
        except Exception as e:
            self.conn.rollback()
            logging.exception(e)

    @contextmanager
    def executemany(self):
        try:
            yield self.cursor.executemany
            self.conn.commit()
        except Exception as e:
            self.conn.rollback()
            logging.exception(e)

    def _tuple_to_object(self, data: List[tuple]) -> List[FetchObject]:
        obj_list = []
        attrs = [desc[0] for desc in self.cursor.description]
        for i in data:
            obj = FetchObject()
            for attr, value in zip(attrs, i):
                setattr(obj, attr, value)
            obj_list.append(obj)
        return obj_list

    def fetchone(self) -> Optional[FetchObject]:
        result = self.cursor.fetchone()
        return self._tuple_to_object([result])[0] if result else None

    def fetchmany(self, size: Optional[int] = None) -> Optional[List[FetchObject]]:
        result = self.cursor.fetchmany(size)
        return self._tuple_to_object(result) if result else None

    def fetchall(self) -> Optional[List[FetchObject]]:
        result = self.cursor.fetchall()
        return self._tuple_to_object(result) if result else None
 Copy code 

Run time type check

Type Hints The name Hints instead of Check, Because it's just a type of prompt, not a real check . Shown above Type Hints usage , In fact, it's all IDE It's helping us complete the function of type checking , But actually ,IDE Type checking does not determine whether an error is reported during code execution , The function of syntax check prompt can only be achieved in the static period .

To implement mandatory type checking during code execution , We need to write our own code or introduce a third-party library （ As described above Pydantic）. Now I pass a type_check The function implements the runtime dynamic check type , For your reference ：

from inspect import getfullargspec
from functools import wraps
from typing import get_type_hints


def type_check(fn):
    @wraps(fn)
    def wrapper(*args, **kwargs):
        fn_args = getfullargspec(fn)[0]
        kwargs.update(dict(zip(fn_args, args)))
        hints = get_type_hints(fn)
        hints.pop("return", None)
        for name, type_ in hints.items():
            if not isinstance(kwargs[name], type_):
                raise TypeError(f"expected {type_.__name__}, got {type(kwargs[name]).__name__} instead")
        return fn(**kwargs)

    return wrapper


# name: str = "world"
name: int = 2

@type_check
def greeting(name: str) -> str:
    return str(name)

print(greeting(name))
# > TypeError: expected str, got int instead
 Copy code 

As long as give greeting The function is marked with type_check Decorator , Run time type checking can be realized .

appendix

If you want to continue to learn more about using Python Type Hints, Here are some open source projects I recommend for your reference ：

• Pydantic [github.com/samuelcolvi…]

• FastAPI [github.com/tiangolo/fa…]

• Tornado [github.com/tornadoweb/…]

• Flask [github.com/pallets/fla…]

• Chia-pool [github.com/Chia-Networ…]

• MySQLHandler [github.com/jianghushin…]

Recommended reading

TypeScript Enumeration Guide

Practical experience sharing ： Use PyO3 To build your Python modular