current position:Home>Using Python to realize multitasking process
Using Python to realize multitasking process
2022-01-29 17:28:02 【Jacko's it journey】
source : official account 【 Jay's IT The journey 】
author : Alaska
ID:Jake_Internet
The article links : utilize Python Implement multitasking process
One 、 Process introduction
process : The program being executed , By procedure 、 Data and process control blocks , Is the program being executed , An execution of a program , yes resources The basic unit of dispatch .
Program : No executed code , It's a static .
Two 、 Comparison between threads and processes
It can be seen from the picture that : At this time, the computer has 9 An application process , But a process will correspond to multiple threads , We can conclude that :\
process : Be able to accomplish multiple tasks , You can run multiple on one computer at the same time QQ
Threads : Be able to accomplish multiple tasks , One QQ Multiple chat windows in
Fundamental difference : Process is the basic unit of operating system resource allocation , Thread is the basic unit of task scheduling and execution .
Advantages of using multiple processes :
1、 Have independence GIL:
First, because in the process GIL The existence of ,Python Multithreading in does not give full play to the advantage of multi-core , Multiple threads in a process , At the same time Only one thread can run at a time . For multiple processes , Every process has its own GIL, therefore , In a multicore processor , Multi process operation is not affected by GIL The impact of . therefore , Progressive Cheng can give better play to the advantages of multi-core .
2、 Efficient
Of course , For reptiles IO For intensive tasks , The effects of multithreading and multiprocessing are not very different . For computing intensive tasks ,Python Multi process compared to multi line cheng , Its multi-core operation efficiency will be doubled .
3、 ... and 、Python Implement multiple processes
Let's use an example to feel :
3.1 Use process class
import multiprocessing
def process(index):
print(f'Process: {index}')
if __name__ == '__main__':
for i in range(5):
p = multiprocessing.Process(target=process, args=(i,))
p.start()
Copy code
This is the most basic way to implement multiple processes : By creating a Process To create a new child process , among target Parameter passed in method name ,args It's the parameters of the method , In order to In the form of tuples , It and the method being called process The parameters of are one-to-one correspondence .
Be careful : here args Must be a tuple , If there is only one parameter , Then add a comma after the first element of the tuple , If there is no comma, then It's no different from a single element itself , Cannot form tuples , Cause problems in parameter passing . After creating the process , We call start Method to start the process .
The operation results are as follows :
Process: 0
Process: 1
Process: 2
Process: 3
Process: 4
Copy code
You can see , We're running 5 Subprocess , Each process calls process Method .process Methodical index Parameters through Process Of args Pass in , Namely 0~4 this 5 A serial number , Finally print it out ,5 The subprocess has finished running .
3.2 Inherit process class
from multiprocessing import Process
import time
class MyProcess(Process):
def __init__(self,loop):
Process.__init__(self)
self.loop = loop
def run(self):
for count in range(self.loop):
time.sleep(1)
print(f'Pid:{self.pid} LoopCount: {count}')
if __name__ == '__main__':
for i in range(2,5):
p = MyProcess(i)
p.start()
Copy code
We first declare a constructor , This method receives a loop Parameters , Represents the number of cycles , And set it as a global variable . stay run In the method , Use this again individual loop The variable loops loop And print the current process number and cycle times .
In the call , We use it range The method gets 2、3、4 Three numbers , And initialize them separately MyProcess process , And then call start Method starts the process Come on .
Be careful : Here, the execution logic of the process needs to be in run Method implementation , To start the process, you need to call start Method , After call run Method will execute .
The operation results are as follows :
Pid:12976 LoopCount: 0
Pid:15012 LoopCount: 0
Pid:11976 LoopCount: 0
Pid:12976 LoopCount: 1
Pid:15012 LoopCount: 1
Pid:11976 LoopCount: 1
Pid:15012 LoopCount: 2
Pid:11976 LoopCount: 2
Pid:11976 LoopCount: 3
Copy code
Be careful , The process here pid Represents the process number , Different machines 、 The results may be different at different times .
Four 、 Communication between processes
4.1 Queue- queue fifo
from multiprocessing import Queue
import multiprocessing
def download(p): # Download data
lst = [11,22,33,44]
for item in lst:
p.put(item)
print(' The data has been downloaded successfully ....')
def savedata(p):
lst = []
while True:
data = p.get()
lst.append(data)
if p.empty():
break
print(lst)
def main():
p1 = Queue()
t1 = multiprocessing.Process(target=download,args=(p1,))
t2 = multiprocessing.Process(target=savedata,args=(p1,))
t1.start()
t2.start()
if __name__ == '__main__':
main()
The data has been downloaded successfully ....
[11, 22, 33, 44]
Copy code
4.2 Shared global variables are not suitable for multiprocess programming
import multiprocessing
a = 1
def demo1():
global a
a += 1
def demo2():
print(a)
def main():
t1 = multiprocessing.Process(target=demo1)
t2 = multiprocessing.Process(target=demo2)
t1.start()
t2.start()
if __name__ == '__main__':
main()
Copy code
Running results :
1
Copy code
It turns out that : Global variables are not shared ;
5、 ... and 、 Communication between process pools
5.1 Process pool introduction
When the number of subprocesses to be created is small , Can be used directly multiprocessing Medium Process Dynamically generate multiple processes , But if there are hundreds or even thousands of goals , The workload of manually creating processes is huge , It can be used at this time multiprocessing Module provided Pool Method .
from multiprocessing import Pool
import os,time,random
def worker(a):
t_start = time.time()
print('%s Start execution , The process number is %d'%(a,os.getpid()))
time.sleep(random.random()*2)
t_stop = time.time()
print(a," Execution completed , Time consuming %0.2f"%(t_stop-t_start))
if __name__ == '__main__':
po = Pool(3) # Define a process pool
for i in range(0,10):
po.apply_async(worker,(i,)) # Add... To the process pool worker The task of
print("--start--")
po.close()
po.join()
print("--end--")
Copy code
Running results :
--start--
0 Start execution , The process number is 6664
1 Start execution , The process number is 4772
2 Start execution , The process number is 13256
0 Execution completed , Time consuming 0.18
3 Start execution , The process number is 6664
2 Execution completed , Time consuming 0.16
4 Start execution , The process number is 13256
1 Execution completed , Time consuming 0.67
5 Start execution , The process number is 4772
4 Execution completed , Time consuming 0.87
6 Start execution , The process number is 13256
3 Execution completed , Time consuming 1.59
7 Start execution , The process number is 6664
5 Execution completed , Time consuming 1.15
8 Start execution , The process number is 4772
7 Execution completed , Time consuming 0.40
9 Start execution , The process number is 6664
6 Execution completed , Time consuming 1.80
8 Execution completed , Time consuming 1.49
9 Execution completed , Time consuming 1.36
--end-
Copy code
A process pool can only hold 3 A process , New tasks cannot be added until the execution is completed , In the process of continuous opening and release .
6、 ... and 、 Case study : Batch copying of files
Operation idea :
- Get the name of the folder to copy
- Create a new folder
- Get all the file names to be copied in the folder
- Create a process pool
- Add tasks to the process pool
The code is as follows :
Guide pack
import multiprocessing
import os
import time
Copy code
Custom file copy function
def copy_file(Q,oldfolderName,newfolderName,file_name):
# File replication , No need to return
time.sleep(0.5)
# print('\r from %s Copy folder to %s The folder %s file '%(oldfolderName,newfolderName,file_name),end='')
old_file = open(oldfolderName + '/' + file_name,'rb') # Files to be copied
content = old_file.read()
old_file.close()
new_file = open(newfolderName + '/' + file_name,'wb') # Copy out the new file
new_file.write(content)
new_file.close()
Q.put(file_name) # towards Q Add file to queue
Copy code
Define the main function
def main():
oldfolderName = input(' Please enter the name of the folder to copy :') # step 1 Get the name of the folder to copy ( You can create... Manually , You can also create... Through code , Here we manually create )
newfolderName = oldfolderName + ' Copy '
# Step two Create a new folder
if not os.path.exists(newfolderName):
os.mkdir(newfolderName)
filenames = os.listdir(oldfolderName) # 3. Get all the file names to be copied in the folder
# print(filenames)
pool = multiprocessing.Pool(5) # 4. Create a process pool
Q = multiprocessing.Manager().Queue() # Create a queue , communicate
for file_name in filenames:
pool.apply_async(copy_file,args=(Q,oldfolderName,newfolderName,file_name)) # 5. Add tasks to the process pool
po.close()
copy_file_num = 0
file_count = len(filenames)
# I don't know when to finish , So define an endless loop
while True:
file_name = Q.get()
copy_file_num += 1
time.sleep(0.2)
print('\r Copy progress %.2f %%'%(copy_file_num * 100/file_count),end='') # Make a copy of the progress bar
if copy_file_num >= file_count:
break
Copy code
The program runs
if __name__ == '__main__':
main()
Copy code
The results are shown in the following figure :
Comparison of file directory structure before and after operation
Before running
After operation
The above is the general result of the whole , because test There are test files pasted casually , There will be no demonstration here .
The end of this paper .
Originality is not easy. , If you think this article is useful to you , Please like this article 、 Comment or forward , Because it will be my motivation to output more quality articles , thank !
by the way , Dig friends remember to give me a free attention ! In case you get lost, you won't find me next time .
See you next time !
copyright notice
author[Jacko's it journey],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201291728009817.html
The sidebar is recommended
- Quickly build Django blog based on function calculation
- Python interface test unittest usage details
- Implementation of top-level design pattern in Python
- You can easily get started with Excel. Python data analysis package pandas (VII): breakdown
- Using linear systems in python with scipy.linalg
- Together with Python to do a license plate automatic recognition system, fun and practical!
- Using linear systems in python with scipy.linalg
- Fast power modulus Python implementation of large numbers
- Quickly build Django blog based on function calculation
- You can easily get started with Excel pandas (I): filtering function
guess what you like
-
You can easily get started with Excel. Python data analysis package pandas (II): advanced filtering (I)
-
How does Python correctly call jar package encryption to get the encrypted value?
-
Python 3 interview question: give an array. If there is 0 in the array, add a 0 after 0, and the overall array length remains the same
-
Python simple Snake game (single player mode)
-
Using linear systems in python with scipy.linalg
-
Python executes functions and even code through strings! Come and understand the operation of such a top!
-
Decoding the verification code of Taobao slider with Python + selenium, the road of information security
-
[Python introduction project] use Python to generate QR code
-
Vanessa basks in her photos and gets caught up in the golden python. There are highlights in the accompanying text. She can't forget Kobe after all
-
[windows] Python installation pyteseract
Random recommended
- [introduction to Python project] create bar chart animation in Python
- Python series tutorials 116
- Practical series 1 ️⃣ Wechat applet automatic testing practice (with Python source code)
- [common links of Python & Python]
- [Python development tool Tkinter designer]: Lecture 1: introduction to the basic functions of Tkinter Designer
- [introduction to Python tutorial] use Python 3 to teach you how to extract any HTML main content
- Python socket implements UDP server and client
- Python socket implements TCP server and client
- leetcode 1974. Minimum Time to Type Word Using Special Typewriter(python)
- The mobile phone uses Python to operate picture files
- [learning notes] Python exception handling try except...
- Two methods of using pandas to read poorly structured excel. You're welcome to take them away
- Python sum (): the summation method of Python