current position:Home>Analysis of Python crawler programming and installation of mitmproxy

Analysis of Python crawler programming and installation of mitmproxy

2022-02-02 10:55:41 cwh5920

mitmproxy It's a support HTTP and HTTPS The package grabber , similar Fiddler、Charles The function of , It's just that it operates in the form of a console .

Besides ,mitmproxy There are also two associated components , One is mitmdump, It is mitmproxy Command line interface for , It can be used to dock Python Script , Realize the processing after listening ; The other is mitmweb, It's a Web Program , Through it to clearly observe mitmproxy Captured requests .

In this section , Let's get to know mitmproxy、mitmdump and mitmweb Installation method of .

  1. Related links

    GitHub:https://github.com/mitmproxy/mitmproxy Official website :https://mitmproxy.orgPyPI:https://pypi.python.org/pypi/mitmproxy Official documents :http://docs.mitmproxy.orgmitmdump Script :http://docs.mitmproxy.org/en/stable/scripting/overview.html Download address :https://github.com/mitmproxy/mitmproxy/releasesDockerHub:https://hub.docker.com/r/mitmproxy/mitmproxy

  2. pip install

The easiest way to install is to use pip, Directly execute the following command to install :

pip3 install mitmproxy

This is the simplest and most versatile way to install , It can be finished after execution mitmproxy Installation , In addition, it is also equipped with mitmdump and mitmweb These two components . If you don't want to install in this way , You can also choose the installation method specifically for each platform listed later or Docker Installation mode .
3. Windows Installation below

You can go to GitHub Upper Releases page ( Link to :https://github.com/mitmproxy/mitmproxy/releases/) Get installation package , Pictured 1-59 Shown .

chart 1-59 The download page

such as , The latest version is 2.0.2, You can choose to download Windows Under the exe Installation package mitmproxy-2.0.2-windows-installer.exe, After downloading, double-click the installation package directly to install .

Be careful , stay Windows I don't support mitmproxy Console interface , But you can use mitmdump and mitmweb.
4. Linux Installation below

stay Linux Next , Download the binary package... OK ( Download address https://github.com/mitmproxy/mitmproxy/releases/), This distribution package is usually the latest version , It contains the latest version of mitmproxy And built-in Python 3 Environmental Science , And the latest OpenSSL Environmental Science .

If you don't have... In your environment Python 3 and OpenSSL Environmental Science , It is recommended to install in this way .

After downloading , You need to unzip and configure it to the environment variable :

tar -zxvf mitmproxy-2.0.2-linux.tar.gz
sudo mv mitmproxy mitmdump mitmweb /usr/bin

In this way, we can 3 Executable files moved to /usr/bin Catalog . And in general ,/usr/bin The directories have been configured in the environment variable , So you can call this directly next 3 A tool .
5. Mac Installation below

Mac The installation under is very simple , Use it directly Homebrew that will do , The order is as follows :

brew install mitmproxy

After executing the command , Can finish mitmproxy Installation .
6. Docker install

mitmproxy Also support Docker, Its DockerHub The address for https://hub.docker.com/r/mitmproxy/mitmproxy/.

stay Docker Next ,mitmproxy The installation command of is :

docker run --rm -it -p 8080:8080 mitmproxy/mitmproxy mitmdump

That's right. 8080 The port is activated mitmproxy and mitmdump.

If you want to get CA certificate , You can choose the mount disk option , The order is as follows :

docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy mitmdump

So you can do it ~/.mitmproxy In the directory CA certificate .

in addition , You can also do it in 8081 Start on port mitmweb, The order is as follows :

docker run --rm -it -p 8080:8080 -p 127.0.0.1:8081:8081 mitmproxy/mitmproxy mitmweb

For more startup methods, please refer to Docker Hub Installation instructions for .
7. Certificate configuration

about mitmproxy Come on , If you want to intercept HTTPS request , You need to set up a certificate .mitmproxy A set of... Will be provided after installation CA certificate , As long as the client trusts mitmproxy Certificate provided , You can go through mitmproxy obtain HTTPS The specific content of the request , otherwise mitmproxy It can't be resolved HTTPS Requested .

First , Run the following command to generate CA certificate , And start the mitmdump:

mitmdump

Next , Then we can in the user directory .mitmproxy Found in the catalog CA certificate , Pictured 1-60 Shown .

chart 1-60 Certificate file

Certificate total 5 individual , surface 1-1 Briefly explain this 5 A certificate .

surface 1-1 5 A certificate and its description
Windows

double-click mitmproxy-ca.p12, The boot page for importing certificates will appear , Pictured 1-61 Shown .

chart 1-61 Certificate Import Wizard

Just click “ next step ” button , A password setting prompt will appear , Pictured 1-62 Shown .

chart 1-62 Password setting prompt

There is no need to set a password here , Just click “ next step ” button .

Next, you need to select the storage area of the certificate , Pictured 1-63 Shown . Click the second option here “ Put all certificates in the following storage ”, And then click “ Browse ” Button , Select the certificate storage location as “ Trusted root certification authority ”, Then click on “ determine ” Button , And then click “ next step ” Button .

chart 1-63 Select the certificate storage area

Last , If a safety warning pops up , Pictured 1-64 Shown , Just click “ yes ” button .

chart 1-64 Safety warning

That's right. Windows Next configuration is finished CA Certificate .
Mac

Mac Double click down mitmproxy-ca-cert.pem The key chain management page will pop up , And then find mitmproxy certificate , Open its setting options , choice “ Always trust ” that will do , Pictured 1-65 Shown .

chart 1-65 Certificate configuration
iOS

take mitmproxy-ca-cert.pem File sent to iPhone On , It is recommended to send by email , And then in iPhone You can directly click on the attachment and identify the installation , Pictured 1-66 Shown .

chart 1-66 Certificate installation page

Click on “ install ” After the button , It will jump to the page of installation description file , Click on “ install ” Button , There will be a warning prompt , Pictured 1-67 Shown .

chart 1-67 Install warning page

Continue to click... In the upper right corner “ install ” Button , After the installation is successful, there will be an installed prompt , Pictured 1-68 Shown .

chart 1-68 Installation success page

If your iOS The version is 10.3 The following words , Trust... Here CA The certificate process has been completed .

If your iOS The version is 10.3 And above , Still need to be in “ Set up ”→“ Universal ”→“ About the machine ”→“ Certificate trust settings ” take mitmproxy The full trust switch is on , Pictured 1-69 Shown . here , stay iOS Configure trust on CA The certificate process is over .

chart 1-69 Certificate trust settings
Android

stay Android On the cell phone , The certificate also needs to be mitmproxy-ca-cert.pem Send files to your phone , For example, copy files directly .

Next , Click Certificate , A prompt window will appear , Pictured 1-70 Shown .

chart 1-70 Certificate installation page

Enter the name of the certificate , And then click “ determine ” Button to complete the installation .

copyright notice
author[cwh5920],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/02/202202021055384276.html

Random recommended