current position:Home>Python crawler actual combat, requests module, python to grab the beautiful wallpaper of a station
Python crawler actual combat, requests module, python to grab the beautiful wallpaper of a station
2022-01-30 19:58:19 【Dai mubai】
「 This is my participation 11 The fourth of the yuegengwen challenge 1 God , Check out the activity details :2021 One last more challenge 」.
Preface
utilize Python Crawl back to the desktop wallpaper , I don't say much nonsense .
Let's start happily ~
development tool
Python edition : 3.6.4
Related modules :
requests modular ;
re modular
As well as some Python Built in modules .
Environment building
install Python And add to environment variable ,pip Install the relevant modules required .
Thought analysis
Target website mm.enterdesk.com/dalumeinv/1…
After entering the website, you can see the following contents from the drop-down menu : Click on any image , Go to the picture details page , Inside is a group of pictures , Contains large images and thumbnails :
This page prohibits the right mouse button , Press ctrl+u View the web page source code , Find the picture link, which can be obtained in the web page source code ; Each picture has two links , Comparing the two links, it is found that one of them has more parameters _360_360, The link without this parameter is the original HD image , The other is the standard definition drawing !
The details page is accessed by the home page link , Let's go back to the home page , Press ctrl+u View the web page source code ; It is found that there is a link to the details page in the web page source code , It can be inferred that both the home page and the detail page are statically loaded web pages !
Drop down the page on the home page , It is found that it will continue to load data , But the website hasn't changed :
But click the page turning operation below alone , The website will change :
thus it can be seen , For page turning operation, we only need to change the parameters of the website :
https://mm.enterdesk.com/dalumeinv/1.html
https://mm.enterdesk.com/dalumeinv/2.html
https://mm.enterdesk.com/dalumeinv/3.html
Copy code
Core code
def main(html_url): # Incoming home page url
response = get_response(html_url) # The request function receives the home page url And request data
urls = re.findall('<a href="(.*?)" target="_blank">.*?</a>', response.text)[31:47] # Extract details page url
for link in urls:
response_ = get_response(link)# Request function receive details page url And request data
image_url = re.findall('src="(https://up.enterdesk.com/edpic/.*?)"', response_.text)[1:] # To extract the image url
url_data(image_url) # Back to picture url
Copy code
Delete selected data
The complete source code can be found in the profile of the personal home page
Data saving local
copyright notice
author[Dai mubai],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201301958172538.html
The sidebar is recommended
- Exploratory data analysis (EDA) in Python using SQL and Seaborn (SNS).
- Turn audio into shareable video with Python and ffmpeg
- Using rbind in python (equivalent to R)
- Pandas: how to create an empty data frame with column names
- Talk about quantifying investment using Python
- Python, image restoration in opencv - CV2 inpaint
- Python notes (14): advanced technologies such as object-oriented programming
- Python notes (13): operations such as object-oriented programming
- Python notes (12): inheritance such as object-oriented programming
- Chapter 2: Fundamentals of python-5 Boolean
guess what you like
-
Python notes (11): encapsulation such as object-oriented programming
-
Python notes (10): concepts such as object-oriented programming
-
Gradient lifting method and its implementation in Python
-
Van * Python | simple crawling of a site course
-
Chapter 1 preliminary knowledge of pandas (list derivation and conditional assignment, anonymous function and map method, zip object and enumerate method, NP basis)
-
Nanny tutorial! Build VIM into an IDE (Python)
-
Fourier transform of Python OpenCV image processing, lesson 52
-
Introduction to python (III) network request and analysis
-
China Merchants Bank credit card number recognition project (Part I), python OpenCV image processing journey, Part 53
-
Introduction to python (IV) dynamic web page analysis and capture
Random recommended
- Python practice - capture 58 rental information and store it in MySQL database
- leetcode 119. Pascal's Triangle II(python)
- leetcode 31. Next Permutation(python)
- [algorithm learning] 807 Maintain the city skyline (Java / C / C + + / Python / go / trust)
- The rich woman's best friend asked me to write her a Taobao double 11 rush purchase script in Python, which can only be arranged
- Glom module of Python data analysis module (1)
- Python crawler actual combat, requests module, python realizes the full set of skin to capture the glory of the king
- Summarize some common mistakes of novices in Python development
- Python libraries you may not know
- [Python crawler] detailed explanation of selenium from introduction to actual combat [2]
- This is what you should do to quickly create a list in Python
- On the 55th day of the journey, python opencv perspective transformation front knowledge contour coordinate points
- Python OpenCV image area contour mark, which can be used to frame various small notes
- How to set up an asgi Django application with Postgres, nginx and uvicorn on Ubuntu 20.04
- Initial Python tuple
- Introduction to Python urllib module
- Advanced Python Basics: from functions to advanced magic methods
- Python Foundation: data structure summary
- Python Basics: from variables to exception handling
- Python notes (22): time module and calendar module
- Python notes (20): built in high-order functions
- Python notes (17): closure
- Python notes (18): decorator
- Python notes (16): generators and iterators
- Python notes (XV): List derivation
- Python tells you what timing attacks are
- Python -- file and exception
- [Python from introduction to mastery] (IV) what are the built-in data types of Python? Figure out
- Python code to scan code to pay attention to official account login
- [algorithm learning] 1221 Split balanced string (Java / C / C + + / Python / go / trust)
- Python notes (22): errors and exceptions
- Python has been hidden for ten years, and once image recognition is heard all over the world
- Python notes (21): random number module
- Python notes (19): anonymous functions
- Use Python and OpenCV to calculate and draw two-dimensional histogram
- Python, Hough circle transformation in opencv
- A library for reading and writing markdown in Python: mdutils
- Datetime of Python time operation (Part I)
- The most useful decorator in the python standard library
- Python iterators and generators