current position：Home>Analysis of Python crawler programming and installation of mitmproxy
Analysis of Python crawler programming and installation of mitmproxy
2022-02-02 10:55:41 【cwh5920】
mitmproxy It's a support HTTP and HTTPS The package grabber , similar Fiddler、Charles The function of , It's just that it operates in the form of a console .
Besides ,mitmproxy There are also two associated components , One is mitmdump, It is mitmproxy Command line interface for , It can be used to dock Python Script , Realize the processing after listening ; The other is mitmweb, It's a Web Program , Through it to clearly observe mitmproxy Captured requests .
In this section , Let's get to know mitmproxy、mitmdump and mitmweb Installation method of .
GitHub：https://github.com/mitmproxy/mitmproxy Official website ：https://mitmproxy.orgPyPI：https://pypi.python.org/pypi/mitmproxy Official documents ：http://docs.mitmproxy.orgmitmdump Script ：http://docs.mitmproxy.org/en/stable/scripting/overview.html Download address ：https://github.com/mitmproxy/mitmproxy/releasesDockerHub：https://hub.docker.com/r/mitmproxy/mitmproxy
The easiest way to install is to use pip, Directly execute the following command to install ：
pip3 install mitmproxy
This is the simplest and most versatile way to install , It can be finished after execution mitmproxy Installation , In addition, it is also equipped with mitmdump and mitmweb These two components . If you don't want to install in this way , You can also choose the installation method specifically for each platform listed later or Docker Installation mode .
3. Windows Installation below
You can go to GitHub Upper Releases page （ Link to ：https://github.com/mitmproxy/mitmproxy/releases/） Get installation package , Pictured 1-59 Shown .
chart 1-59 The download page
such as , The latest version is 2.0.2, You can choose to download Windows Under the exe Installation package mitmproxy-2.0.2-windows-installer.exe, After downloading, double-click the installation package directly to install .
Be careful , stay Windows I don't support mitmproxy Console interface , But you can use mitmdump and mitmweb.
4. Linux Installation below
stay Linux Next , Download the binary package... OK （ Download address https://github.com/mitmproxy/mitmproxy/releases/）, This distribution package is usually the latest version , It contains the latest version of mitmproxy And built-in Python 3 Environmental Science , And the latest OpenSSL Environmental Science .
If you don't have... In your environment Python 3 and OpenSSL Environmental Science , It is recommended to install in this way .
After downloading , You need to unzip and configure it to the environment variable ：
tar -zxvf mitmproxy-2.0.2-linux.tar.gz
sudo mv mitmproxy mitmdump mitmweb /usr/bin
In this way, we can 3 Executable files moved to /usr/bin Catalog . And in general ,/usr/bin The directories have been configured in the environment variable , So you can call this directly next 3 A tool .
5. Mac Installation below
Mac The installation under is very simple , Use it directly Homebrew that will do , The order is as follows ：
brew install mitmproxy
After executing the command , Can finish mitmproxy Installation .
6. Docker install
mitmproxy Also support Docker, Its DockerHub The address for https://hub.docker.com/r/mitmproxy/mitmproxy/.
stay Docker Next ,mitmproxy The installation command of is ：
docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy mitmdump
That's right. 8080 The port is activated mitmproxy and mitmdump.
If you want to get CA certificate , You can choose the mount disk option , The order is as follows ：
docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy mitmdump
So you can do it ~/.mitmproxy In the directory CA certificate .
in addition , You can also do it in 8081 Start on port mitmweb, The order is as follows ：
docker run --rm -it -p 8080:8080 -p 127.0.0.1:8081:8081 mitmproxy/mitmproxy mitmweb
For more startup methods, please refer to Docker Hub Installation instructions for .
7. Certificate configuration
about mitmproxy Come on , If you want to intercept HTTPS request , You need to set up a certificate .mitmproxy A set of... Will be provided after installation CA certificate , As long as the client trusts mitmproxy Certificate provided , You can go through mitmproxy obtain HTTPS The specific content of the request , otherwise mitmproxy It can't be resolved HTTPS Requested .
First , Run the following command to generate CA certificate , And start the mitmdump：
Next , Then we can in the user directory .mitmproxy Found in the catalog CA certificate , Pictured 1-60 Shown .
chart 1-60 Certificate file
Certificate total 5 individual , surface 1-1 Briefly explain this 5 A certificate .
surface 1-1 5 A certificate and its description
double-click mitmproxy-ca.p12, The boot page for importing certificates will appear , Pictured 1-61 Shown .
chart 1-61 Certificate Import Wizard
Just click “ next step ” button , A password setting prompt will appear , Pictured 1-62 Shown .
chart 1-62 Password setting prompt
There is no need to set a password here , Just click “ next step ” button .
Next, you need to select the storage area of the certificate , Pictured 1-63 Shown . Click the second option here “ Put all certificates in the following storage ”, And then click “ Browse ” Button , Select the certificate storage location as “ Trusted root certification authority ”, Then click on “ determine ” Button , And then click “ next step ” Button .
chart 1-63 Select the certificate storage area
Last , If a safety warning pops up , Pictured 1-64 Shown , Just click “ yes ” button .
chart 1-64 Safety warning
That's right. Windows Next configuration is finished CA Certificate .
Mac Double click down mitmproxy-ca-cert.pem The key chain management page will pop up , And then find mitmproxy certificate , Open its setting options , choice “ Always trust ” that will do , Pictured 1-65 Shown .
chart 1-65 Certificate configuration
take mitmproxy-ca-cert.pem File sent to iPhone On , It is recommended to send by email , And then in iPhone You can directly click on the attachment and identify the installation , Pictured 1-66 Shown .
chart 1-66 Certificate installation page
Click on “ install ” After the button , It will jump to the page of installation description file , Click on “ install ” Button , There will be a warning prompt , Pictured 1-67 Shown .
chart 1-67 Install warning page
Continue to click... In the upper right corner “ install ” Button , After the installation is successful, there will be an installed prompt , Pictured 1-68 Shown .
chart 1-68 Installation success page
If your iOS The version is 10.3 The following words , Trust... Here CA The certificate process has been completed .
If your iOS The version is 10.3 And above , Still need to be in “ Set up ”→“ Universal ”→“ About the machine ”→“ Certificate trust settings ” take mitmproxy The full trust switch is on , Pictured 1-69 Shown . here , stay iOS Configure trust on CA The certificate process is over .
chart 1-69 Certificate trust settings
stay Android On the cell phone , The certificate also needs to be mitmproxy-ca-cert.pem Send files to your phone , For example, copy files directly .
Next , Click Certificate , A prompt window will appear , Pictured 1-70 Shown .
chart 1-70 Certificate installation page
Enter the name of the certificate , And then click “ determine ” Button to complete the installation .
author[cwh5920],Please bring the original link to reprint, thank you.
The sidebar is recommended
- Teach you three Python Programming Tips
- Python code reading (Chapter 69): case conversion of initial letters
- Python tutorial series 133 -- several special grammars
- Dry goods | Python operation: Python controls Excel to realize office automation
- Understand the principle of affine transformation and python implementation code
- A little trick every day, python can easily convert PDF to text and bid farewell to copy and paste
- Climb Conan barrage with Python + gephi to sort out the main plot
- As a programmer, do you know Python variable reference
- One line of code, take you to learn Python
- Summary of 22 advanced Python knowledge points, dry goods!
guess what you like
[Python learning] nanny level teaching parsing and parsing XML in Python
Use of Python JSON module
"Python" guide to using itertools of Python standard library
What are the functions and classifications of Python interpreters
Python implements four schemes for timed tasks, lazy artifact
Python can actually realize the freedom of punch in?
What about Python memory leak? Pit filling troubleshooting tips
Object oriented programming in Python
Quick start - Python Basics
[target detection (8)] a thorough understanding of the loss function of the regression box of target detection -- the principles of IOU, giou, Diou and ciou and Python code
- 【Python】2. Logic control statement, list, tuple, dictionary
- Write an Enigma machine in Python
- Python lists, dictionaries, tuples, collections, learning notes
- Python actual combat case, requests module, python implementation, obtaining dynamic charts
- Information recommendation platform based on Python search engine and recommendation algorithm
- It's better to know the python decorator than to know the heart of your girlfriend
- Use Python to develop a dinosaur running game and play it
- Use SciPy FFT for Fourier transform: Python signal processing
- Let the python program automatically play Sudoku, and the second becomes the strongest brain!
- Python ETL tool
- Vue3 + ant design Vue + Python realizes the combined deployment of a set of code and multiple sub applications
- Python snail sorting
- Generating AI web applications using Python and flask
- Share 4 practical Python automation scripts
- [Python basics] Python collaboration
- 11 "Python dictionary" knowledge points easy to use
- These 20 pandas functions can improve your 'data cleaning' ability by 100 times
- British one-to-one student Python poker homework Q & A
- Django's 35000 word blog post
- Four skills of pandas row column conversion
- Python actual combat case, mplcyberpunk module, python realizes cyberpunk data visualization
- What are the differences between golang and python 3?
- Detailed explanation of Python + fidder 4 web crawler (the most complete in Station C)
- Pycham operation and configuration guide can't do this. It must be slow to write python
- "Top level introduction to Python" is the devil's step step by step
- Summary of the most complete Python Library in Station C - standard library + advanced library (10000 word long text, recommended Collection)
- Python realizes GIF dynamic drawing and video card communication, kicks two feet to break the dimensional wall | machine learning
- Python realizes the cartoon of photos and breaks the dimensional wall with one punch | machine learning
- Python to achieve picture beautification, drunk do not know the sky in the water? (attach code) | machine learning
- Python projects are packaged into executable exe files
- Python video conversion resolution (with code) | Python tool
- Python modify picture resolution (with code) | Python tool
- Python file upload Minio and Ali OSS tools | Python tools
- Python video add audio (with code) | Python tool
- Python game development, python to achieve Snake games and eat beans with source code
- What language is Python
- [untitled] Python magic method__ new__ method
- Use Python to get the information of popular positions and see what skills you need from a high salary
- Notes on Python deep learning (11): loss function
- Python foundation 2.4 -- tuple