current position:Home>Python multi process realizes the word frequency statistics function of document library under MapReduce model

Python multi process realizes the word frequency statistics function of document library under MapReduce model

2022-02-02 12:02:27 Cachel wood

 Insert picture description here

import time
import jieba
from multiprocessing import Process,Manager

def data(path):  #data Used to extract document data , Summarize to list 
    lis = []
    file = open(path,'r',encoding='gb18030')
    txt = file.readline()
    n = 1
    for txt in file:
        if n%6==4:
            content = txt.strip('\n')
            content = content.replace('<content>','') # Remove useless information 
            content = content.replace('</content>','')
            if len(content)>0:
                lis.append(content)
        n+=1
    file.close()
    return lis  # All documents are summarized into a list and returned 

def Map(path,lis):  #Map Function to segment words and store them in the list 
    for pa in path:
        text_lis = jieba.lcut(pa)
        for i in text_lis:
            lis.append((i,1))
    #print(len(lis))

def Reduce(lis):  #Reduce Function to summarize the results into a dictionary 
    print('time2 = %f'%(time.time()-start_time))  # test Map Function takes time ( Word segmentation always takes time )
    dic = {
    }
    for k,v in lis:
        dic[k] = dic.get(k,0)+1
        #print(dic)
    dic_order=sorted(dic.items(),key=lambda x:x[1],reverse=True)  # Sort dictionary in descending order 
    with open('data.txt','w',encoding='utf-8') as file:
        for k,v in dic_order:
            file.write(k+':'+str(v)+'\n')  # Write the results to a file 

if __name__=='__main__':
    start_time = time.time()
    path = data('news_sohusite_xml.csv')
    print('time1 = %f'%(time.time()-start_time))  # Test the time taken to extract documents 
    plist = []
    m = Manager()
    list1 = m.list([])
    for i in range(10):   # Create a process 
        p = Process(target=Map,args=(path[10000*i:10000*(i+1)],list1))
        plist.append(p)	
    for p in plist:
        p.start()  # Start the process 
    for p in plist:
        p.join()  # Block the main process 
    Reduce(list1) # When Map When the process is complete Reduce Reduce the results 
    time = time.time() - start_time
    print('time3 = %f'%(time))   # Total test time 
    print('main')
time1 = 21.858970
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 1.170 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 1.592 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.133 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.086 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.370 seconds.
Prefix dict has been built successfully.
Loading model cost 2.410 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.318 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.303 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.327 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.474 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.355 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.748 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.061 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.551 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 1.838 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 1.865 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.109 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.597 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.558 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.337 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.162 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.297 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.331 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.218 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.180 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.236 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.026 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.266 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.474 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.485 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.093 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.073 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.763 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.388 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.477 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.776 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.645 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.495 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.881 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.701 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.262 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.053 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.275 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.186 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.651 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.337 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.230 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.315 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.445 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.433 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.601 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.796 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.871 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.802 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.878 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.746 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.072 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.766 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.357 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.180 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.232 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.495 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.594 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.441 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.490 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.915 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.805 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.709 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.426 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.483 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.385 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.406 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.399 seconds.
Prefix dict has been built successfully.
Loading model cost 2.903 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.829 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.945 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.468 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.802 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.420 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.627 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.642 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.862 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.633 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.710 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.879 seconds.
Prefix dict has been built successfully.
Loading model cost 3.393 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.213 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.814 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.744 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.379 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.082 seconds.
Prefix dict has been built successfully.
Loading model cost 2.815 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.948 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.503 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.915 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.159 seconds.
Prefix dict has been built successfully.
Loading model cost 2.861 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 3.060 seconds.
Prefix dict has been built successfully.
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\lenovo\AppData\Local\Temp\jieba.cache
Loading model cost 2.896 seconds.
Prefix dict has been built successfully.
Loading model cost 2.229 seconds.
Prefix dict has been built successfully.
time2 = 518.363330
time3 = 531.016773
main

time1 Is the time to read the document ,time2 It's time to form a complete dictionary ,time3 Is the total time after adding the read file .
 Insert picture description here
By observing the relationship between the number of threads and running time , It can be found that the running time is shorter when the number of threads is small .

copyright notice
author[Cachel wood],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/02/202202021202240788.html

Random recommended