current position:Home>Python crawler native code learning (I)

Python crawler native code learning (I)

2022-01-30 21:41:45 Cooper_ wwj

< Affirming >: The article is my original , For quotation or commercial use, please contact me , Otherwise, we will be held responsible


At present, reptile technology has developed more and more mature , In order to speed up the climbing efficiency and shorten the time , There have been some problems such as “scrapy”、“PySpider” And so on , There have even been excellent “ octopus ” And other integrated software .

These excellent frameworks have some disadvantages for learners , They don't know what the underlying architecture looks like , The specific content is not clear , Only use , Make mistakes and don't know where the problem is , Then I was annoyed there for a long time , This is quite uncomfortable for learners .

that , What does native code look like ?

With this question , I did a series of operations myself , Found a multi-threaded way to climb the whole station . Here you are welcome to comment and give advice .

Software configuration environment

A good hunter often has a good gun in his hand , So do programmers . Here we use python3.9.2 The compiler and pycharm Integrated software .

pycharm.pngpython 3.9.2.png

Learning process

We are mainly divided into five parts to learn the process of crawler infrastructure , Namely :

( One ) Page analysis : Analyze the target page through developer tools , And determine how to write the crawler program
( Two ) Send a request : utilize urllib Library or requests The library initiates a request for the target page , Get response data
( 3、 ... and ) Analyze the response data and get the content : utilize re library 、xpath Library or BeautifulSoup Library to get the required content
( Four ) Store content : utilize sqlite3 library 、excel Worksheet 、word Document or Mysql Libraries store content permanently
( 5、 ... and ) The optimization process :1: Library selection ;2: Enable multithreading ;3: To reprocess ;

flow chart


copyright notice
author[Cooper_ wwj],Please bring the original link to reprint, thank you.

Random recommended