current position:Home>Introduction to python (III) network request and analysis
Introduction to python (III) network request and analysis
2022-01-30 13:20:13 【baiyuliang】
Install network request module :Requests
pip install requests
Do you seem to know each other ? Do you think of nodejs?
A simple test :
First import requests modular :
import requests
Copy code
get request :
response = requests.get("https://www.baidu.com")
print(response)
Copy code
result :
Indicates that the request has been successful , We can view... In the editor response What's in it :
Print response.text:
This is the content of Baidu home page , But it's garbled , take it easy , Add this step :
response = requests.get("https://www.baidu.com")
response.encoding = response.apparent_encoding
print(response.text)
Copy code
OK! Is it often simple ?
Of course ,requests Not just for get, It also supports post,put,delete etc. :
When using different request methods , It also supports headler,param wait :
requests.get("https://www.baidu.com",headers=,params=)
Copy code
This is essential to the network request framework !
We use 360 Picture interface as an example , Make paging requests :wallpaper.apc.360.cn/index.php?c…
Of course , We can go straight through get Request the link , It can also be done through post Request and pass in parameters :
params = {
'c': 'WallPaperAndroid',
'a': 'getAppsByCategory',
'cid': 9,
'start': 0,
'count': 10
}
response = requests.post("http://wallpaper.apc.360.cn/index.php", params=params)
print(response.text)
Copy code
Request the results (json Format ):
analysis json:
json_data = json.loads(response.text)
print('errno=%s,errmsg=%s' % (json_data['errno'], json_data['errmsg']))
list = json_data['data']
print("count=" + str(len(list)))
Copy code
result :
Be careful :print Print log when , Use... After the string + When splicing parameters, only string types can be used , So we need to use str() take int Type to string type !
Okay ,json The parsing of the format is finished , What should I do if I want to parse the web page ? I used... A long time ago java When parsing a web page , Use a tool called jsoup, I believe many students have used , When it is, press xml Format parsing , Various node,element...
python There are similar and powerful web page parsing tools : BeautifulSoup.( notes :python Primary zone xml-sax,xml-dom Parser , But whether it's good or not needs your own experience !)
BeautifulSoup Advantages and disadvantages :
We usually analyze web page data , It's the second kind of :BeautifulSoup(markup, "lxml")!
install BeautifulSoup:pip install bs4
A simple test , Take Baidu home page as an example :
from bs4 import BeautifulSoup
response = requests.get("https://www.baidu.com")
response.encoding = response.apparent_encoding
print(response.text)
soup = BeautifulSoup(response.text, "lxml")
title = soup.find(name='title').text # You can omit it name:soup.find('title')
print(title)
Copy code
Error report in execution :
Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library? Copy code
Solution :
1. install virtualenv: pip install virtualenv
1. install lxml: pip install lxml
Re execution py Program , result :
copyright notice
author[baiyuliang],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201301320113205.html
The sidebar is recommended
- Some people say Python does not support function overloading?
- "Python instance" was shocked and realized the dirty words and advertisement detection of the chat system with Python
- Introduction to Python - CONDA common commands
- Python actual combat | just "4 steps" to get started with web crawler (with benefits)
- Don't know what to eat every day? Python to tell you! Generate recipes and don't worry about what to eat every day!
- Are people who like drinking tea successful? I use Python to make a tea guide! Do you like it?
- I took 100g pictures offline overnight with Python just to prevent the website from disappearing
- Binary operation of Python OpenCV image re learning and image smoothing (convolution processing)
- Analysis of Python event mechanism
- Iterator of Python basic language
guess what you like
-
Base64 encryption and decryption in Python
-
Chapter 2: Fundamentals of python-2 variable
-
Python garbage collection summary
-
Python game development, pyGame module, python takes you to realize a magic tower game from scratch (1)
-
Python draws a spinning windmill with turtle
-
Deep understanding of Python features
-
A website full of temptations for Python crawler writers, "lovely picture network", look at the name of this website
-
Python opencv Canny edge detection knowledge supplement
-
Complex learning of Python opencv Sobel operator, ScHARR operator and Laplacian operator
-
Python: faker extension package
Random recommended
- Python code reading (Part 44): find the location of qualified elements
- Elegant implementation of Django model field encryption
- 40 Python entry applet
- Pandas comprehensive application
- Chapter 2: Fundamentals of python-3 character string
- Python pyplot draws a parallel histogram, and the x-axis value is displayed in the center of the two histograms
- [Python crawler] detailed explanation of selenium from introduction to actual combat [1]
- Curl to Python self use version
- Python visualization - 3D drawing solutions pyecharts, Matplotlib, openpyxl
- Use python, opencv's meanshift and CAMSHIFT algorithms to find and track objects in video
- Using python, opencv obtains and changes pixels, modifies image channels, and trims ROI
- [Python data collection] university ranking data collection
- [Python data collection] stock information collection
- Python game development, pyGame module, python takes you to realize a magic tower game from scratch (2)
- Python solves the problem of suspending execution after clicking the mouse in CMD window (fast editing mode is prohibited)
- [Python from introduction to mastery] (II) how to run Python? What are the good development tools (pycharm)
- Python type hints from introduction to practice
- Python notes (IX): basic operation of dictionary
- Python notes (8): basic operations of collections
- Python notes (VII): definition and use of tuples
- Python notes (6): definition and use of lists
- Python notes (V): string operation
- Python notes (IV): use of functions and modules
- Python notes (3): conditional statements and circular statements
- Python notes (II): lexical structure
- Notes on python (I): getting to know Python
- [Python data structure series] - tree and binary tree - basic knowledge - knowledge point explanation + code implementation
- [Python daily homework] Day7: how to combine two dictionaries in an expression?
- How to implement a custom list or dictionary in Python
- 15 advanced Python tips for experienced programmers
- Python string method tutorial - how to use the find() and replace() functions on Python strings
- Python computer network basics
- Python crawler series: crawling global airport information
- Python crawler series: crawling global port information
- How to calculate unique values using pandas groupby
- Application of built-in distribution of Monte Carlo simulation SciPy with Python
- Gradient lifting method and its implementation in Python
- Pandas: how to group and calculate by index
- Can you create an empty pandas data frame and fill it in?
- Python basic exercises teaching! can't? (practice makes perfect)