current position:Home>My friend's stock suffered a terrible loss. When I was angry, I crawled the latest data of securities with Python

My friend's stock suffered a terrible loss. When I was angry, I crawled the latest data of securities with Python

2022-01-30 20:06:05 Hairy programmer

Grab the target :

url:http://webapi.cninfo.com.cn/#/marketDataDate
 Data target :  obtain   Stock code   Securities abbreviation   Transaction date    Opening price   Highest price   The lowest price   Closing price   Number of deals 
 difficulty :  Request header parameters mcode  Encryption generation  
 Copy code 

Using third party libraries :

1. requests
2. execjs
3. js2py
4. math
5. time
6. pandas
 Copy code 

Tools :

1.  Google browser 
2. pycharm
3. python3.7
 Copy code 

Page analysis :

Enter the page , Right mouse click , Select Check -> Pop up browser developer tools -> Netword -> choice xhr -> Refresh the page

  Find the interface , Observe the interface data :

This interface is post request , Two form parameters need to be submitted .

code snippet :

url = 'http://webapi.cninfo.com.cn/api/sysapi/p_sysapi1015'
data = {
    'tdate': datetime,
    'scode': '399001'
}
headers = {
    'mcode': 'MTYzNTEzOTkxMQ==',
    'Referer': 'http://webapi.cninfo.com.cn/',
    'Cookie': 'Hm_lvt_489bd07e99fbfc5f12cbb4145adb0a9b=1634795282; Hm_lpvt_489bd07e99fbfc5f12cbb4145adb0a9b=1634799860',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
}
response = requests.post(url, headers=headers, data=data).json()
 Copy code 

The request is successful , But there is a problem of timeliness , After a few minutes , You have no access , Careful observation shows that special parameters mcode.

Start JS Find the encryption parameter generation location : ( Note that you need to refresh the page , Otherwise, there will be no js situation )

  The problem is coming. , several js file , How to make sure that's what we need ?

Use the second search method , according to url Search for the requested address :

 

After hitting the breakpoint Refresh the page directly Start debugging

var indexcode={
    getResCode:function(){
        var time=Math.floor(new Date().getTime()/1000); return window.JSonToCSV.missjson(""+time);
    }
}
 Copy code 

Continue debugging , Look at this. missjson What are the .

  Let's take these js Take it down and execute

Complete code

import requests
import execjs
import js2py
import math
import time
import pandas as pd
​
​
code_list = []
​
​
def MCODE():
    jscode = '''
    function missjson(input) {  
        var keyStr = "ABCDEFGHIJKLMNOP" + "QRSTUVWXYZabcdef" + "ghijklmnopqrstuv"   + "wxyz0123456789+/" + "=";  
        var output = "";  
        var chr1, chr2, chr3 = "";  
        var enc1, enc2, enc3, enc4 = "";  
        var i = 0;  
        do {  
            chr1 = input.charCodeAt(i++);  
            chr2 = input.charCodeAt(i++);  
            chr3 = input.charCodeAt(i++);  
            enc1 = chr1 >> 2;  
            enc2 = ((chr1 & 3) << 4) | (chr2 >> 4);  
            enc3 = ((chr2 & 15) << 2) | (chr3 >> 6);  
            enc4 = chr3 & 63;  
            if (isNaN(chr2)) {  
                enc3 = enc4 = 64;  
            } else if (isNaN(chr3)) {  
                enc4 = 64;  
            }  
            output = output + keyStr.charAt(enc1) + keyStr.charAt(enc2)  
                    + keyStr.charAt(enc3) + keyStr.charAt(enc4);  
            chr1 = chr2 = chr3 = "";  
            enc1 = enc2 = enc3 = enc4 = "";  
        } while (i < input.length);  
    
        return output;  
    } 
    
    '''
    time1 = js2py.eval_js('Math.floor(new Date().getTime()/1000)')
    # py The way 
    a = math.floor(time.time() / 1000)
    mcode = execjs.compile(jscode).call('missjson', '{a}'.format(a=time1))
    return mcode
​
​
def PageRquest(datetime, mcode):
    #  The interface can be changed 
    url = 'http://webapi.cninfo.com.cn/api/sysapi/p_sysapi1015'
    data = {
        'tdate': datetime, #  Time to get data 
        'scode': '399001'  #  Stock code   And exchange abbreviation 
    }
    headers = {
        'mcode': str(mcode),
        'Referer': 'http://webapi.cninfo.com.cn/',
        'Cookie': 'Hm_lvt_489bd07e99fbfc5f12cbb4145adb0a9b=1634795282; Hm_lpvt_489bd07e99fbfc5f12cbb4145adb0a9b=1634799860',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
    }
    response = requests.post(url, headers=headers, data=data).json()
    code = response['records']
    for i in code:
        code_list.append(i)
​
​
def main(date):
    mcode = MCODE()
    PageRquest(date, mcode)
​
​
if __name__ == '__main__':
    # main()
    #  Data analysis  pandas  Automated office 
    datetime = pd.period_range('2021/5/1', '2021/10/20', freq='B')
    for date in datetime:
        main(date)
    df = pd.DataFrame(code_list)
    df.to_excel('code.xlsx')
 Copy code 

copyright notice
author[Hairy programmer],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201302006043280.html

Random recommended