Analysis of the principle of Python import

One 、 brief introduction

Python code in one module gains access to the code in another module by the process of importing it.

Simply speaking , What we see everyday .py file , Is called a module.

When your python The code needs to get some external functions （ Some wheels have been built ）, You need to use import This declaration keyword .import Can help import other module .（ similar C An appointment include）

import Declaration is a common import method , But it's not the only way . That is, it can be done in other ways module Import .

import Statement combines two operations ：

1. Search for named modules . Call... By passing in the appropriate parameters __import()__ Realization .
2. Bind the search results to the local namespace . __import()__ The return value of is used as the namespace binding operation .

import Statement execution time ,__import__() Will be called ,Python I'll look for module And create a module object And initialize it ; If module Did not find , Will throw out ModuleNotFoundError One of the exception .

import vs __import__()

Simply speaking , call __import__() It's just import Declare a subset of operations .

Call directly import() Perform module search only , If you find , Execute the module creation operation . Although there may be some side effects , For example, import parent package , And updating various caches （ Include sys.modules）, But only import Statement to perform name binding operation .

Two 、 lookup Module The way

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:The directory containing the input script (or the current directory when no file is specified). PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH). The installation-dependent default.

import Execution time , Will try to find... In the following order module：

1. The parser first tries to search for its own built-in module
2. If you can't find it , Will be based on sys.path Order lookup for
   1. py The folder where the execution file itself is located ;
   2. PYTHONPATH environment variable ;
   3. python The default installation depends on the location

You can view through the following operations sys.path The path of

$ python3
Python 3.5.2 (default, Jan 26 2021, 13:30:48) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/tester/opt/2.7.5.1', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages']

therefore , If you need to use some third-party libraries , In addition to pip Call directly after installation , Also can put the module Put it in the corresponding directory , And then use PYTHONPATH Specify the directory .

3、 ... and 、import Lead in deep exploration

• Import requires module Complete path ,Python Will try to import from top to bottom . When python Try importing foo.bar.baz when , Will try to import foo , then foo.bar, Last foo.bar.baz, If any intermediate import fails , Will trigger ModuleNotFoundError.
• Import yes cache The concept of . Each import will try to sys.modules This cache Search for , If you return None It will be thrown out. ModuleNotFoundError error , If module name Can't find ,Python Will try to keep looking down
• import The latest underlying mechanism of , It's through finders and loaders Combine the two to find module And import ,finders Responsible for finding relevant paths , loaders Responsible for loading .

actual module Lookup order

Python’s default sys.meta_path has three meta path finders, one that knows how to import built-in modules, one that knows how to import frozen modules, and one that knows how to import modules from an import path (i.e. the path based finder).

As can be seen from the above ,python import It will follow a certain search order . Except for the second chapter 3 A path , There is another layer in the actual front cache

1. sys.modules,The module cache.
2. sys.meta_path: Generally speaking, there are 3 individual finders
   1. one that knows how to import built-in modules
   2. one that knows how to import frozen modules
   3. one that knows how to import modules from an import path

this 3 individual finders The corresponding path , Just as the second chapter above said 3 A search path

Can pass Python Interactive command to view

[email protected][SJC]~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.meta_path)
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]

Four 、 Relative Import and absolute import

Relative Import , With . Lead , Follow linux File systems are similar to ,. and .. They represent the present package And the parent package

Absolutely import , It's usually import <> or from <> import <> This form

for example ：

package The directory structure is as follows

package/
    __init__.py
    subpackage1/
        __init__.py
        moduleX.py
        moduleY.py
    subpackage2/
        __init__.py
        moduleZ.py
    moduleA.py

The following are relative imports

from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY
from ..subpackage1 import moduleY
from ..subpackage2.moduleZ import eggs
from ..moduleA import foo

Here is the absolute import

import package.subpackage1.moduleX
from package.subpackage1 import moduleX

5、 ... and 、 Extension

Generally we use import Import module when , What principles should be followed ,PEP8 The following suggestions are given ：

Imports should be grouped in the following order:Standard library imports. Related third party imports. Local application/library specific imports. You should put a blank line between each group of imports.

import Organizational order ：

1. Standard library import
2. Third party Library import
3. Local application or library import

importlib

Python The new version provides a api Can be controlled import The rules of , Avoid changing directly in the old way __import__() Complicated operation , And reduce the concept of errors .

importlib Modules provide a wealth of API To interact with the import system . for example importlib.import_module() Provides a recommended 、 Than built-in __import__() Simpler API To invoke the import mechanism .

Searching

sys.modules Will be in module import Update when done cache, For next import Quick access .

sys.modules, The module cache This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths. So if foo.bar.baz was previously imported, sys.modules will contain entries for foo, foo.bar, and foo.bar.baz. Each key will have as its value the corresponding module object.

finder My job is to search , Follow loader Separation of work

A finder’s job is to determine whether it can find the named module using whatever strategy it knows about.

Search path , It's not just sys.path All the paths , some subpackages Your search may depend on parent package Of __path__.

import path: A list of locations (or path entries) that are searched by the path based finder for modules to import. During import, this list of locations usually comes from sys.path, but for subpackages it may also come from the parent package’s __path__ attribute.

import The mechanism is extensible , Detailed view Import hooks The concept . There are two main import hooks: meta hooks and import path hooks

The import machinery is designed to be extensible; the primary mechanism for this are the import hooks. There are two types of import hooks: meta hooks and import path hooks.

finders

3 Default finder, Search for different policies module

Python’s default sys.meta_path has three meta path finders, one that knows how to import built-in modules, one that knows how to import frozen modules, and one that knows how to import modules from an import path (i.e. the path based finder).

Reference resources

1. https://docs.python.org/3/reference/import.html
2. https://docs.python.org/3/tutorial/modules.html#the-module-search-path
3. https://www.mediumcn.com/python3/what-happens-behind-the-scenes-when-we-import-a-module-in-python
4. http://sinhub.cn/2019/05/python-import-machinery-part-two/
5. https://stackoverflow.com/questions/9586630/python-paths-and-import-order