apache2.4 + django + windows10 Automated Deployment

stay django Build a... In the project webhooks url route , The automated deployment script is executed asynchronously under this route When not deployed , Access this route There is no problem Successfully executed the script

The script involves two things Pull update local code and restart apache2.4

apache2.4 Deploy djano after , No problem with project access , But when executing Automated Deployment scripts The following problems have been encountered

1. Can't find the virtual environment python The path of the interpreter , sys.executable The return is httpd.exe The path of
2. Cannot introduce... Into automation scripts django Package files in the project , Only the current python Package file in environment !
3. The environment variables executed in the automation script are different than expected %USERPROFILE% route Change into C:\WINDOWS\system32\config\systemprofile

apache2.4 After deployment Cannot find current python The path of the interpreter

1. Bug Reappear

    #  stay  django  Output in any view  sys.executable
    def test(request)   """    test   View    """      logger.info("sys.executable: {}".format(sys.executable))  Copy code 

   And then restart apache Access the view You'll be surprised Because it outputs : c:\apache24\bin\httpd.exe

   why????

   With doubts I printed it directly sys.path Try find python route

   But its output is In virtual environment Package file path and Project root path

   like this : e:\development\django_project\env\Lib\site-packages

2. Solution

   ok I can still find the current through this path python The path of use \Scripts\python.exe Replace the following \Lib\site-packages

In automated scripts git pull Command error

The content of the error report is as follows

Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

Say I don't have authority , How is that possible? I just push 了 After Baidu It seems that the problem comes down to .ssh Catalog problems

Generally speaking git When you pull the code Will establish ssh Connect that ssh seek ssh_key When Will look for from the current user directory .ssh Catalog Which corresponds to key

therefore I printed it directly this time %USERPROFILE% Value

Sure enough , stay apache After deployment %USERPROFILE% Turned into C:\WINDOWS\system32\config\systemprofile

ah ! this ? I can't help it , I don't know where to start , Hence the Current user's .shh Catalog Made a copy to The path above goes

restart apache The script runs normally

In automated scripts introduce django Custom package file for the project

My hands are cheap and optimized Deployment scripts

Extract some keys to conf\env.py Inside Unified management

Manual start django project Perfect operation adopt apache After deployment There was no response

I then added logger Find out Import env When you report an error, you can't find This package file

It's just !!! ok Can only adopt How to open a file load Key

This time, Perfect operation !!!

subprocess.py Report errors

Ecstatic To test

git commit -m "fix: Test deployment script "

git push

then I waited for a while No response I opened the update log and looked

Wrong report :

 [2021-11-14 02:27:01,467][root.<module>():119] [ERROR] Traceback (most recent call last):
   File "E:\development\django_project\utils\auto_update.py", line 117, in <module>
     update()
   File "E:\development\django_project\utils\auto_update.py", line 88, in update
     flag, git_res = git_pull()
   File "E:\development\django_project\utils\auto_update.py", line 55, in git_pull
     messages = exec_cmd(message_list)
   File "E:\development\django_project\utils\util.py", line 54, in exec_cmd
     out, err = proc.communicate(timeout=timeout)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\Lib\subprocess.py", line 964, in communicate
     stdout, stderr = self._communicate(input, endtime, timeout)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\Lib\subprocess.py", line 1317, in _communicate
     stdout = stdout[0]
 IndexError: list index out of range
 ​
 Copy code 

The relevant code is as follows : Comment part I added it later

 # django_project\utils\auto_update.py
 from pip._internal import main as pip
 def exec_cmd(cmd: str, timeout=10):
     proc = subprocess.Popen(cmd,
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.STDOUT,
                             encoding='gbk')
     res = []
     try:
         out, err = proc.communicate(timeout=timeout)
     except subprocess.TimeoutExpired as ex:
         res.append("commend_timeout:{}".format(ex))
         return res
     for line in out.splitlines():
         res.append(line)
 ​
     return res
 ​
 def git_pull():
     """         git pull     :return:     """
     pull = 'git -C {} pull --force origin master:master'.format(project_path)
     change_file_list = 'git -C {} diff --name-status [email protected]{{1}}..HEAD'.format(project_path)
     message_list = 'git -C {} log [email protected]{{1}}..HEAD --pretty=format:"%s from %cn at %ci"'.format(project_path)
     res_lines = exec_cmd(pull, 20)
     logging.info(res_lines)
     res_lines.insert(0, '<span style="color:red;">1. git pull</span>')
     if 'Already up to date.' != res_lines[-1]:
         return False, res_lines
     change_file = exec_cmd(change_file_list, 20)
     logging.info(change_file)
     change_file.insert(0, '<span style="color:red;">2. change file list</span>')
     res_lines.extend(change_file)
     # time.sleep(1) 
     # try:
     messages = exec_cmd(message_list, 20)
     # logging.info(messages)
     messages.insert(0, '<span style="color:red;">3. push message</span>')
     res_lines.extend(messages)
     # except Exception:
     #   logging.error('message:  Acquisition failure : {}'.format(traceback.format_exc()))
     for item in change_file:
         if 'requirments.txt' in item:
             pip(['install', '-r', os.path.join(project_path, 'requirments.txt'), '-i',
                  'https://pypi.douban.com/simple'])
             break
     return True, res_lines
 ​
 Copy code 

I then added the comment section logger Find out Such an output [2021-11-14 02:54:18,058][root.git_pull():60] [INFO] ['style: �����°��Զ����� from bangenlanbai at 2021-11-14 02:17:20 +0800']

chinese ??? commit ? How did the decoding fail . proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, encoding='gbk')

ok in utf-8 try , After the change It is found that there is no error reported here , however below adopt net stop apache2.4 and net start apache2.4 Decoding will fail again

therefore This function I changed it to

 def exec_cmd(cmd: str, timeout=10):
     proc = subprocess.Popen(cmd,
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.STDOUT,
                             encoding=None) #  here   Don't pass on encoding  It will return   Byte type   Output 
     res = []
     try:
         out, err = proc.communicate(timeout=timeout)
     except subprocess.TimeoutExpired as ex:
         res.append("commend_timeout:{}".format(ex))
         return res
     for line in out.splitlines():
         # line  here   yes  bytes  type 
         try:
             line_str = str(line, encoding='GBK')  #  Try coding here 
         except Exception:
             line_str = str(line, encoding='utf-8')
         res.append(res)
     return res
 Copy code 

After a change The script runs perfectly !

There's a little problem

python Default logging to file There will be GBK Coding problem

 #  Default log configuration   File format   yes  GBK
 logging.basicConfig(level=logging.INFO,
                     filename=log_path,
                     filemode='a',
                     format=
                     '[%(asctime)s][%(name)s.%(funcName)s():%(lineno)d] [%(levelname)s] %(message)s',
                     )
 ​
 #  Use  logging.FileHandler(filename=log_path, encoding='utf-8')  Specify the file code   There will be no problem 
 logger = logging.getLogger('auto_update')
 file_handel = logging.FileHandler(filename=log_path, encoding='utf-8')
 logger.setLevel('INFO')
 file_handel.setFormatter(
     logging.Formatter('[%(asctime)s][%(name)s.%(funcName)s():%(lineno)d] [%(levelname)s] %(message)s'))
 logger.addHandler(file_handel)

 Copy code 

The improved deployment script is as follows

 # -*- coding:utf-8 -*-
 # @Time     : 2021/11/12 19:32
 # @Author   : BGLB
 # @Software : PyCharm
 # auto_update.py 
 ​
 import datetime
 import logging
 import os
 import subprocess
 import sys
 import time
 import traceback
 ​
 from dingtalkchatbot.chatbot import DingtalkChatbot
 from pip._internal import main as pip
 ​
 self_path = os.path.abspath(__file__)
 ​
 project_path = os.path.dirname(os.path.dirname(self_path))
 log_path = os.path.join(project_path, 'logs', 'auto_update.log')
 ​
 logger = logging.getLogger('auto_update')
 file_handel = logging.FileHandler(filename=log_path, encoding='utf-8')
 logger.setLevel('INFO')
 file_handel.setFormatter(
     logging.Formatter('[%(asctime)s][%(name)s.%(funcName)s():%(lineno)d] [%(levelname)s] %(message)s'))
 logger.addHandler(file_handel)
 ​
 ​
 def exec_cmd(cmd: str, timeout=10):
     proc = subprocess.Popen(cmd,
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.STDOUT,
                             encoding=None)
     res = []
     try:
         out, err = proc.communicate(timeout=timeout)
     except subprocess.TimeoutExpired as ex:
         res.append("commend_timeout:{}".format(ex))
         return res
     for line in out.splitlines():
         try:
             line_str = line.decode()
         except Exception:
             line_str = line.decode('GBK')
 ​
         res.append(line_str)
 ​
     return res
 ​
 ​
 def start_auto_update():
     """          Start yourself      :return:     """
     python_path = sys.executable if sys.executable.endswith('python.exe') else sys.path[0].replace('Lib\site-packages',
                                                                                                    'Scripts\python.exe')
     os.system('start "auto_update" /d {} {} {}'.format(project_path, python_path, self_path))
 ​
 ​
 def git_pull():
     """         git pull     :return:     """
     pull = 'git -C {} pull --force origin master:master'.format(project_path)
     change_file_list = 'git -C {} diff --name-status [email protected]{{1}}..HEAD'.format(project_path)
     message_list = 'git -C {} log [email protected]{{1}}..HEAD --pretty=format:"%s from %cn at %ci"'.format(project_path)
     res_lines = exec_cmd(pull, 20)
     logger.info(res_lines)
     res_lines.insert(0, '<span style="color:red;">1. git pull</span>')
     if 'Already up to date.' != res_lines[-1]:
         return False, res_lines
     change_file = exec_cmd(change_file_list, 20)
     logger.info(change_file)
     change_file.insert(0, '<span style="color:red;">2. change file list</span>')
     res_lines.extend(change_file)
     time.sleep(1)
     messages = exec_cmd(message_list, 20)
     logging.info(messages)
     messages.insert(0, '<span style="color:red;">3. push message</span>')
     res_lines.extend(messages)
     for item in change_file:
         if 'requirments.txt' in item:
             try:
                 logger.info(' download  requirments.txt')
                 pip(['install', '-r', os.path.join(project_path, 'requirments.txt'), '-i',
                      'https://pypi.douban.com/simple'])
                 break
             except Exception:
                 log_str = f" Download failed ：\n{traceback.format_exc()}"
                 logger.error(log_str)
                 res_lines.append(log_str)
                 return False, res_lines
     return True, res_lines
 ​
 ​
 def restart_apache():
     """          restart  apache     :return:     """
 ​
     cmd_line = ['net stop apache2.4', 'net start apache2.4']
     res = []
     for cmd in cmd_line:
         res.extend(exec_cmd(cmd, 60*2))
         time.sleep(1)
     if 'apache2.4  Service started successfully .' in res:
         return True, res
     return False, res
 ​
 ​
 def update():
     """          to update      :return:     """
     # from conf.env import DINGDING_ROBOT_URL, DINGDING_SECRET, DINGDING_AT_MOBILES
     DINGDING_ROBOT_URL = ''
     DINGDING_SECRET = ''
     DINGDING_AT_MOBILES = []
     #  Read here env  The configuration is still a little inelegant 
     env_path = os.path.join(project_path, 'conf', 'env.py')
     with open(env_path, encoding='utf=8', mode='r') as f:
         text = f.readlines()
     for item in text:
         if item and item.startswith('DINGDING'):
             value = item.strip(' ').strip('\n').split('=', 1)[-1]
 ​
             if 'DINGDING_ROBOT_URL' in item:
                 DINGDING_ROBOT_URL = eval(value)
 ​
             if item and 'DINGDING_SECRET' in item:
                 DINGDING_SECRET = eval(value)
 ​
             if item and 'DINGDING_AT_MOBILES' in item:
                 DINGDING_AT_MOBILES = eval(value)
 ​
     dingding = DingtalkChatbot(DINGDING_ROBOT_URL, secret=DINGDING_SECRET)
     flag, git_res = git_pull()
     markdown = '#  Automated Deployment logs \n\nstart_time: {}\n\n'.format(datetime.datetime.now().strftime('%Y-%m-%d ''%H:%M:%S'))
     markdown += '\n\n**pull update**\n'
     for index, item in enumerate(git_res):
         prefix = '' if item.endswith('</span>') else '> - '
         markdown += '\n{}{}\n'.format(prefix, item.lstrip())
 ​
     if not flag:
         markdown += '\n\n\n**<span style="color: red">git pull  Failure </span>  Go and have a look **\n'
         dingding.send_markdown(title=' Automated Deployment failed ', text=markdown, at_mobiles=DINGDING_AT_MOBILES)
         return
 ​
     flag, restart_apache_res = restart_apache()
     markdown += '\n\n\n**restart apache**\n>\n'
     for item in restart_apache_res:
         if item.strip(' '):
             markdown += '> - {}\n'.format(item.lstrip(' '))
 ​
     if not flag:
         markdown += '\n\n\n**<span style="color: red">restart apache  Failure </span>  Go and have a look **\n\n'
         dingding.send_markdown(title=' Automated Deployment failed ', text=markdown, at_mobiles=DINGDING_AT_MOBILES)
         return
 ​
     markdown += '\n\n\nend_time: {}\n\n\n'.format(datetime.datetime.now().strftime('%Y-%m-%d ''%H:%M:%S'))
     dingding.send_markdown(title=' Automated Deployment successful ', text=markdown, at_mobiles=DINGDING_AT_MOBILES)
 ​
 #  Nail robot  markdown  grammar   It doesn't seem to support  html  grammar 
 if __name__ == '__main__':
     try:
         update()
     except Exception:
         logger.error(traceback.format_exc())
 ​
 Copy code 

Project structure

 ─django_project
     │  manage.py
     │  requirments.txt
     ├─conf
     │  │  env.py
     ├─utils
     │  │  auto_update.py
     ├─logs
     │  │  ...
     ├─django_project
     │  │  setting.py
     ├─test_app
     │  │  view.py
     │  │  models.py
     │  │  urls.py
 Copy code