current position:Home>Some people say Python does not support function overloading?

Some people say Python does not support function overloading?

2022-01-30 06:11:56 somenzz

as everyone knows ,Python It's dynamic language , The so-called dynamic language , Is that the type of variable is dynamic , Variable types can be changed at will during program operation , because Python There is no fixed type of variable , Function overloading depends on variable types , Overloading is to define multiple functions with the same name , But the parameter types of these functions with the same name are different , When different types of parameters are passed in, the corresponding functions are executed .

Python There is no type for the variable , therefore Python The syntax itself does not support function overloading , So someone said Python Function overloading is not supported. This statement itself is correct , But what this article wants to say is ,Python The flexibility of dynamic language can realize multiple functions of a function without function overloading . But let Python Function overloading is really supported , Specifically, there are two schemes .

Scheme 1 、 Pseudo overloading

Java The advantage of that overload is that you can see which variable types the function supports from the form of the function , and Python Because variables have no fixed type , This is not very readable , For example, the following function fun, In fact, it supports two parameters , One is all strings , One is that all are integers , But without looking at the code , In fact, it can't be seen :

def fun(x, y):
    if isinstance(x, str) and isinstance(y, str):
        print(f"str {x =}, {y = } ")
    elif isinstance(x, int) and isinstance(y, int):
        print(f"int {x = }, {y = }")
fun("hello", "world")
fun(1,2)
 Copy code 

Running results

str x ='hello', y = 'world' 
int x = 1, y = 2
 Copy code 

But fortunately, Python Type prompt , With the help of Python The standard library of typing, We can also write code in overloaded form :

import typing


class A:
 @typing.overload
    def fun(self, x: str, y: str) -> None:
        pass

 @typing.overload
    def fun(self, x: int, y: int) -> None:
        pass

    def fun(self, x, y) -> None:
        if isinstance(x, str) and isinstance(y, str):
            print(f"str {x =}, {y = } ")
        elif isinstance(x, int) and isinstance(y, int):
            print(f"int {x = }, {y = }")


if __name__ == "__main__":
    a = A()
    a.fun("hello", "world")
    a.fun(1, 2)

 Copy code 

Running results :

str x ='hello', y = 'world' 
int x = 1, y = 2
 Copy code 

In this case , Readability is improved , But this is a formal overload , What really works is the last function without decorator , The first two functions with decorators exist only for better readability , There is no practical effect , You can delete , Does not affect the operation of the program .

If you want to achieve Java That's a real function overload , Please see plan 2 .

Option two , With the help of a metaclass , Achieve true overloading

Metaclasses are Python More advanced features , If you give the complete code at the beginning , You may not understand , Here, the implementation process is shown step by step .

Python Everything is an object , for instance 1 yes int Example ,int yes type example :

In [7]: a = 5

In [8]: type(a)
Out[8]: int

In [9]: type(int)
Out[9]: type

In [10]:
In [11]: type??
Init signature: type(self, /, *args, **kwargs)
Docstring:
type(object_or_name, bases, dict)
type(object) -> the object's type type(name, bases, dict) -> a new type Type: type Subclasses: ABCMeta, EnumMeta, _TemplateMetaclass, _ABC, MetaHasDescriptors, NamedTupleMeta, _TypedDictMeta, LexerMeta, StyleMeta, _NormalizerMeta, ...  Copy code 

As can be seen from the above ,type(object) return object The type of , and type(name, bases, dict) A new type will be generated , in other words type(name, bases, dict) Will produce a class:

In [17]: A = type('A',(),{})

In [18]: a = A()

In [19]: type(a)
Out[19]: __main__.A
In [20]: type(A)
Out[20]: type
 Copy code 

The above code , amount to :

In [21]: class A:
    ...:     pass
    ...:

In [22]: a = A()

In [23]: type(a)
Out[23]: __main__.A

In [24]: type(A)
Out[24]: type
 Copy code 

I understand that , Even if you don't use class keyword , We can also create a class to , For example, the following make_A() and A() It's the same thing :

class A:
    a = 1
    b = "hello"
    def fun(self):
        return "Class A"

def make_A():
    name = 'A'
    bases = ()
    a = 1
    b = "hello"

    def fun():
        return "Class A"

    namespace = {'a':a,'b':b,'fun': fun}

    return type(name,bases,namespace)

if __name__ == '__main__':
    a = A()
    print(a.b)
    print(a.fun())
    print("==="*5)
    b = make_A()
    print(b.b)
    print(b.fun())

 Copy code 

Execution results :

hello
Class A
===============
hello
Class A
 Copy code 

Please note that make_A There is a in the function namespace, It's a dictionary , It stores the member variables and member functions of the class , When we define multiple functions with the same name in a class , The last one will cover all the front , This is the characteristic of a dictionary , The same key is assigned multiple times , Only the last one will be kept , therefore Python Class does not support function overloading .

Now we need to keep multiple functions with the same name , Then rewrite the dictionary , When the same key is assigned multiple times , Put these values ( function ) Keep in a list , Write a class with specific methods , Inherit dict, Then write the code as follows :

class OverloadDict(dict):
    def __setitem__(self, key, value):
        assert isinstance(key, str), "keys must be str"

        prior_val = self.get(key, _MISSING)
        overloaded = getattr(value, "__overload__", False)

        if prior_val is _MISSING:
            insert_val = OverloadList([value]) if overloaded else value
            super().__setitem__(key, insert_val)
        elif isinstance(prior_val, OverloadList):
            if not overloaded:
                raise ValueError(self._errmsg(key))
            prior_val.append(value)
        else:
            if overloaded:
                raise ValueError(self._errmsg(key))
            super().__setitem__(key, value)

 @staticmethod
    def _errmsg(key):
        return f"must mark all overloads with @overload: {key}"

 Copy code 

The above code has a key place , That is, if there is overload identification , Then put it on the list prior_val in :

elif isinstance(prior_val, OverloadList):
    if not overloaded:
        raise ValueError(self._errmsg(key))
    prior_val.append(value)

 Copy code 

among OverloadList It's a list , Its definition is as follows :

class OverloadList(list):
    pass
 Copy code 

Write another decorator , Identifies whether a function should be overloaded :

def overload(f):
    f.__overload__ = True
    return 
 Copy code 

Then let's test this OverloadDict, See what it does :

print("OVERLOAD DICT USAGE")
d = OverloadDict()

@overload
def f(self):
    pass

d["a"] = 1
d["a"] = 2
d["b"] = 3
d["f"] = f
d["f"] = f
print(d)
 Copy code 

Running results :

OVERLOAD DICT USAGE
{'a': 2, 'b': 3, 'f': [<function overload_dict_usage.<locals>.f at 0x7fdec70090d0>, <function overload_dict_usage.<locals>.f at 0x7fdec70090d0>]}
 Copy code 

OverloadDict Solved the problem of how to save duplicate name functions , Just put them in a list , There's another problem that hasn't been solved , That is how to get the correct function from the list to execute ?

It must be based on the parameter type passed in by the function , How to achieve it ? With the help of Python Type prompt and introspection module inspect, Yes, of course , And with the help of Python The metaclass of :

class OverloadMeta(type):
 @classmethod
    def __prepare__(mcs, name, bases):
        return OverloadDict()

    def __new__(mcs, name, bases, namespace, **kwargs):
        overload_namespace = {
            key: Overload(val) if isinstance(val, OverloadList) else val
            for key, val in namespace.items()
        }
        return super().__new__(mcs, name, bases, overload_namespace, **kwargs)
 Copy code 

There's a Overload class , Its function is to map the signature and definition of a function , When we use a.f Is called __get__ Method to obtain the corresponding function . Its definition is as follows :


class Overload:
    def __set_name__(self, owner, name):
        self.owner = owner
        self.name = name

    def __init__(self, overload_list):
        if not isinstance(overload_list, OverloadList):
            raise TypeError("must use OverloadList")
        if not overload_list:
            raise ValueError("empty overload list")
        self.overload_list = overload_list
        self.signatures = [inspect.signature(f) for f in overload_list]

    def __repr__(self):
        return f"{self.__class__.__qualname__}({self.overload_list!r})"

    def __get__(self, instance, _owner=None):
        if instance is None:
            return self
        # don't use owner == type(instance)
        # we want self.owner, which is the class from which get is being called
        return BoundOverloadDispatcher(
            instance, self.owner, self.name, self.overload_list, self.signatures
        )

    def extend(self, other):
        if not isinstance(other, Overload):
            raise TypeError
        self.overload_list.extend(other.overload_list)
        self.signatures.extend(other.signatures)
 Copy code 

__get__ Back to a BoundOverloadDispatcher class , It binds the parameter type to the corresponding function , Only when the function is called __call__ Return the most matching function to call :


class BoundOverloadDispatcher:
    def __init__(self, instance, owner_cls, name, overload_list, signatures):
        self.instance = instance
        self.owner_cls = owner_cls
        self.name = name
        self.overload_list = overload_list
        self.signatures = signatures

    def best_match(self, *args, **kwargs):
        for f, sig in zip(self.overload_list, self.signatures):
            try:
                bound_args = sig.bind(self.instance, *args, **kwargs)
            except TypeError:
                pass  # missing/extra/unexpected args or kwargs
            else:
                bound_args.apply_defaults()
                # just for demonstration, use the first one that matches
                if _signature_matches(sig, bound_args):
                    return f

        raise NoMatchingOverload()

    def __call__(self, *args, **kwargs):
        try:
            f = self.best_match(*args, **kwargs)
        except NoMatchingOverload:
            pass
        else:
            return f(self.instance, *args, **kwargs)

        # no matching overload in owner class, check next in line
        super_instance = super(self.owner_cls, self.instance)
        super_call = getattr(super_instance, self.name, _MISSING)
        if super_call is not _MISSING:
            return super_call(*args, **kwargs)
        else:
            raise NoMatchingOverload()
            
def _type_hint_matches(obj, hint):
    # only works with concrete types, not things like Optional
    return hint is inspect.Parameter.empty or isinstance(obj, hint)


def _signature_matches(sig: inspect.Signature, bound_args: inspect.BoundArguments):
    # doesn't handle type hints on *args or **kwargs
    for name, arg in bound_args.arguments.items():
        param = sig.parameters[name]
        hint = param.annotation
        if not _type_hint_matches(arg, hint):
            return False
    return True

 Copy code 

It's almost here , Let's assemble the above code , You can make Python Achieve true overloading :

import inspect

class NoMatchingOverload(Exception):
    pass

_MISSING = object()

class A(metaclass=OverloadMeta):
 @overload
    def f(self, x: int):
        print("A.f int overload", self, x)

 @overload
    def f(self, x: str):
        print("A.f str overload", self, x)

 @overload
    def f(self, x, y):
        print("A.f two arg overload", self, x, y)


class B(A):
    def normal_method(self):
        print("B.f normal method")

 @overload
    def f(self, x, y, z):
        print("B.f three arg overload", self, x, y, z)

    # works with inheritance too!


class C(B):
 @overload
    def f(self, x, y, z, t):
        print("C.f four arg overload", self, x, y, z, t)


def overloaded_class_example():
    print("OVERLOADED CLASS EXAMPLE")

    a = A()
    print(f"{a=}")
    print(f"{type(a)=}")
    print(f"{type(A)=}")
    print(f"{A.f=}")

    a.f(0)
    a.f("hello")
    # a.f(None) # Error, no matching overload
    a.f(1, True)
    print(f"{A.f=}")
    print(f"{a.f=}")

    b = B()
    print(f"{b=}")
    print(f"{type(b)=}")
    print(f"{type(B)=}")
    print(f"{B.f=}")
    b.f(0)
    b.f("hello")
    b.f(1, True)
    b.f(1, True, "hello")
    # b.f(None) # no matching overload
    b.normal_method()

    c = C()
    c.f(1)
    c.f(1, 2, 3)
    c.f(1, 2, 3, 4)
    # c.f(None) # no matching overload


def main():
    overloaded_class_example()


if __name__ == "__main__":
    main()

 Copy code 

The operation results are as follows :


OVERLOADED CLASS EXAMPLE
a=<__main__.A object at 0x7fbabe67d8e0>
type(a)=<class '__main__.A'>
type(A)=<class '__main__.OverloadMeta'>
A.f=Overload([<function A.f at 0x7fbabe679280>, <function A.f at 0x7fbabe679310>, <function A.f at 0x7fbabe6793a0>])
A.f int overload <__main__.A object at 0x7fbabe67d8e0> 0
A.f str overload <__main__.A object at 0x7fbabe67d8e0> hello
A.f two arg overload <__main__.A object at 0x7fbabe67d8e0> 1 True
A.f=Overload([<function A.f at 0x7fbabe679280>, <function A.f at 0x7fbabe679310>, <function A.f at 0x7fbabe6793a0>])
a.f=<__main__.BoundOverloadDispatcher object at 0x7fbabe67d910>
b=<__main__.B object at 0x7fbabe67d910>
type(b)=<class '__main__.B'>
type(B)=<class '__main__.OverloadMeta'>
B.f=Overload([<function B.f at 0x7fbabe6794c0>])
A.f int overload <__main__.B object at 0x7fbabe67d910> 0
A.f str overload <__main__.B object at 0x7fbabe67d910> hello
A.f two arg overload <__main__.B object at 0x7fbabe67d910> 1 True
B.f three arg overload <__main__.B object at 0x7fbabe67d910> 1 True hello
B.f normal method
A.f int overload <__main__.C object at 0x7fbabe67d9a0> 1
B.f three arg overload <__main__.C object at 0x7fbabe67d9a0> 1 2 3
C.f four arg overload <__main__.C object at 0x7fbabe67d9a0> 1 2 3 4
 Copy code 

The code is long , Put together is not conducive to reading and understanding , But all the code is shown in the body , If you don't want to assemble it yourself , Just want complete one click code , Can pay attention to 「Python Number seven 」, Dialog reply 「 heavy load 」 Get the implementation Python Overloaded complete code .

Last words

This overload efficiency is bound to be very low , You are not recommended to use such overloads at all (Python It doesn't need to be overloaded , Overloading is a matter of static language ), Just an example with the help of implementation overload , Make you understand Python The metaclass of , That is to say metaclass The power of , In fact, the implementation idea can be used for reference , Through this paper , Can you get More flexible programming skills , My goal has been achieved , If it helps , Please have a look at , Or forward it to someone who needs it , Thank you for reading .

Pay attention to me , Learn a little skill every day .

copyright notice
author[somenzz],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201300611548137.html

Random recommended