current position:Home>Go Python 3 usage and pit Prevention Guide

Go Python 3 usage and pit Prevention Guide

2022-02-01 03:53:43 Call me cousin

go-python3 Journey of exploration

brief introduction

​ I haven't updated my blog for a long time , The author has graduated for more than a year , Working in a large factory in China , Recently, I received a challenging job in the process of work , In the process of solving, it is found that there is still a lack of online learning materials in this regard , So I specially recorded . you 're right , Is the use of go call python3 Some practice and exploration of . as everyone knows ,go It's a static type of language ,python It's a dynamic type of language , To call a dynamic language on a statically typed language , The translation work during this period is actually very complicated , There is a lot of uncertainty , Now let's take a look at the course of this exploration .

Environment building

​github There are ready-made go call python3 Open source framework ,github.com/DataDog/go-… It is worth noting that at present, this library only supports python3.7, When I first started, the computer was equipped with python3.8, So I spent a lot of time on this , But it also taught me a lesson , When using an open source library, you must take a good look at it readme.md, Notice what's worth noting .

 Insert picture description here

​ Use go get github.com/DataDog/go-python3 You can get this library into your project , But it may also report an error , If it is pkg_config You can install this dependency on your computer ,mac Installation on :brew install pkg_config , Do it again , The problem is solved .

Program demo

​ The author's program directory structure is like this :  Insert picture description here

​ Let's look at the specific use of this library demo:

1.go Code test_python3.go

package main

import (
   "fmt"
   "github.com/DataDog/go-python3"
   "log"
   "os"
)

func ImportModule(dir, name string) *python3.PyObject {
   sysModule := python3.PyImport_ImportModule("sys")
   path := sysModule.GetAttrString("path")
   pathStr, _ := pythonRepr(path)
   log.Println("before add path is " + pathStr)
   python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(""))
   python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
   pathStr, _ = pythonRepr(path)
   log.Println("after add path is " + pathStr)
   return python3.PyImport_ImportModule(name)
}

func main(){
   python3.Py_Initialize()
   if !python3.Py_IsInitialized() {
      fmt.Println("Error initializing the python interpreter")
      os.Exit(1)
   }
   path, _ := os.Getwd()
   helloPy := ImportModule(path + "/py3", "test_python3")
   if helloPy == nil {
      log.Fatalf("helloPy is nil")
      return
   }
   helloFunc := helloPy.GetAttrString("test_print_name")
   if helloFunc == nil {
      log.Fatalf("helloFunc is nil")
   }
   var args = python3.PyTuple_New(1)
   python3.PyTuple_SetItem(args, 0, python3.PyUnicode_FromString("python3"))
   helloPy3Str := helloFunc.Call(args, python3.Py_None)
   if helloPy3Str == nil {
      log.Fatalf("helloPy3Str is nil")
   }
   funcResultStr, _ := pythonRepr(helloPy3Str)
   log.Println("func result: " + funcResultStr)
}

func pythonRepr(o *python3.PyObject) (string, error) {
   if o == nil {
      return "", fmt.Errorf("object is nil")
   }

   s := o.Repr()
   if s == nil {
      python3.PyErr_Clear()
      return "", fmt.Errorf("failed to call Repr object method")
   }
   defer s.DecRef()

   return python3.PyUnicode_AsUTF8(s), nil
}
 Copy code 

2.python Code test_python3.py

import os

from test_import_py import f
from test_import_py import x

# import cython
def test_print_name(name):
    print(os.getcwd())
    y = f(1.1)
    print(x)
    return y
 Copy code 

3.python&cython Code

import cython

x = 3

@cython.cfunc
@cython.exceptval(-2, check=True)
def f(x: cython.double) -> cython.double:
    return x ** 2 - x
 Copy code 

Program run results

 Insert picture description here

Program test point

  1. How to introduce and execute a python Method ?
  2. stay python There will be no problem introducing other modules into the code
  3. stay python Write... In the program Cython Will there be a problem

Result analysis

  1. python3.Py_Initialize() Call this method to initialize python3 Execution environment , Only when this method is executed can the related python3 operation , It's worth noting that this method has no return value , Nor can it be initialized repeatedly , Repeated initialization will report an error . This method has no return value, which means that we can't perceive whether there is a problem in its execution process from this method , But he also offers python3.Py_IsInitialized() This way to let us know python3 Whether the initialization of the environment is complete . These two methods should be used together .

  2. Add search path , We are going to sys.path Add what we wrote python The directory where the code is located is used as the search path , After adding it , The code in the file can be used as a module import Come in . It is worth noting that , When we import When it's a document , Will take this. py The code execution side of the script , Such as py In code print("hello") It's executed

  3. Execute code , Some of them api And noteworthy points are as follows :

    • PyImport_ImportModule: Introduce a module , Popular understanding is to pass in a file name ( The premise is that the directory where the file name is located has been introduced into the search path , That is to say sys.path below ), It is worth noting that , If the imported file path is wrong or does not exist , This method returns zero nil, No errors are reported , If the report is wrong , That should be in the document we introduced python There is a problem with the code , The author encountered a pit , Namely python In the code import Some of the dependencies are not used in the code , And then it's wrong .

    • GetAttrString: Get a variable in the module according to the name , Back to a pythonObject, For example, we will a.py A is defined in demo = 1 So this time we a.GetAttrString("demo") You get this variable , Of course, you can also get a method here .

    • Call: Call a method , It is worth noting that the transmission parameter here is a tuple, If not tuple,Call Method will return nil. Another point of concern is , There is a concurrency problem , If a function is in Call During execution , Called again , here python Environment will crash, I think this should be python Why ,python Of GIL Mechanism , Can't support real concurrent running code . This demo Will be posted on the back .

    • pythonRepr, Encapsulated code , Will a python Object to string .

The number of concurrent method calls

test_concurrency.go

package main

import (
   "fmt"
   "github.com/DataDog/go-python3"
   "log"
   "os"
)

func main()  {
   python3.Py_Initialize()
   if !python3.Py_IsInitialized() {
      fmt.Println("Error initializing the python interpreter")
      os.Exit(1)
   }
   sysModule := python3.PyImport_ImportModule("sys")
   path := sysModule.GetAttrString("path")
   //pathStr, _ := pythonRepr2(path)
   //log.Println("before add path is " + pathStr)
   python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(""))
   python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("./py3"))
   concurrencyFile := python3.PyImport_ImportModule("test_concurrency")
   if concurrencyFile == nil {
      log.Fatalf("concurrency is nil")
      return
   }
   testFunc := concurrencyFile.GetAttrString("test_func")
   if testFunc == nil {
      log.Fatalf("testFunc is nil")
      return
   }
   go func() {
      testFunc.Call(python3.Py_None, python3.Py_None)
      for  {

      }
   }()
  	//time.Sleep(10 * time.Second)
   go func() {
      testFunc.Call(python3.Py_None, python3.Py_None)
      for  {

      }
   }()
   select {

   }
}
 Copy code 

test_concurrency.py

import time

def test_func():
    print("hello")
    time.sleep(10)
 Copy code 
  • The general meaning of this program is python The method is time-consuming 10s( The execution is artificial sleep 10s), And then in go There are two coroutines in the to call this method concurrently , The result of this program is python crash, But if you add the code in the comments , That is to say, add... Between the two processes 11s Deep sleep , These two processes are important for this func The call is serial , So there's no problem .

summary

​ In fact, looking back , This exploration process is still very interesting , Of course, it's great to have such an experience at work . When I adjusted the program, I was excited to sing a song on the station ,hhhh.

​ Come back , The author doesn't quite understand python, So some things are crossing the river by touching stones , A lot of conclusions have a hint of speculation in them , If there is no support, please mention more .

copyright notice
author[Call me cousin],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/02/202202010353392028.html

Random recommended