< Affirming >： The article is my original , For quotation or commercial use, please contact me , Otherwise, we will be held responsible

introduction

At present, reptile technology has developed more and more mature , In order to speed up the climbing efficiency and shorten the time , There have been some problems such as “scrapy”、“PySpider” And so on , There have even been excellent “ octopus ” And other integrated software .

These excellent frameworks have some disadvantages for learners , They don't know what the underlying architecture looks like , The specific content is not clear , Only use , Make mistakes and don't know where the problem is , Then I was annoyed there for a long time , This is quite uncomfortable for learners .

that , What does native code look like ？

With this question , I did a series of operations myself , Found a multi-threaded way to climb the whole station . Here you are welcome to comment and give advice .

Software configuration environment

A good hunter often has a good gun in his hand , So do programmers . Here we use python3.9.2 The compiler and pycharm Integrated software .

Learning process

We are mainly divided into five parts to learn the process of crawler infrastructure , Namely ：

（ One ） Page analysis ： Analyze the target page through developer tools , And determine how to write the crawler program

（ Two ） Send a request ： utilize urllib Library or requests The library initiates a request for the target page , Get response data

（ 3、 ... and ） Analyze the response data and get the content ： utilize re library 、xpath Library or BeautifulSoup Library to get the required content

（ Four ） Store content ： utilize sqlite3 library 、excel Worksheet 、word Document or Mysql Libraries store content permanently

（ 5、 ... and ） The optimization process ：1： Library selection ;2： Enable multithreading ;3： To reprocess ;

flow chart