current position:Home>Solve the problem that Django celery beat prompts that the database is incorrectly configured and does not support multiple databases
Solve the problem that Django celery beat prompts that the database is incorrectly configured and does not support multiple databases
2022-01-31 00:13:21 【Shanxiao I】
1. Problem recurrence
Write a... One day celery When the task is scheduled , start-up celery The rear is ready to start celery-beat, Suddenly found that a large number of warnings pop up and quit the process , Looking up, I found an error
celery beat v5.1.2 (sun-harmonics) is starting.
[2021-11-01 18:32:11,917: INFO/MainProcess] beat: Starting...
[2021-11-01 18:32:12,071: CRITICAL/MainProcess] beat raised exception <class 'django.core.exceptions.ImproperlyConfigured'>: ImproperlyConfigured('settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.')
Traceback (most recent call last):
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
service.start()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 636, in start
humanize_seconds(self.scheduler.max_interval))
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/kombu/utils/objects.py", line 29, in __get__
return super().__get__(instance, owner)
File "/Users/systemime/.pyenv/versions/3.9.6/lib/python3.9/functools.py", line 969, in __get__
val = self.func(instance)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 680, in scheduler
return self.get_scheduler()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 671, in get_scheduler
return symbol_by_name(self.scheduler_cls, aliases=aliases)(
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 231, in __init__
Scheduler.__init__(self, *args, **kwargs)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 271, in __init__
self.setup_schedule()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 239, in setup_schedule
self.install_default_entries(self.schedule)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 340, in install_default_entries
self.update_from_dict(entries)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 328, in update_from_dict
self.schedule.update(s)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 355, in schedule
elif self.schedule_changed():
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 261, in schedule_changed
transaction.commit()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/transaction.py", line 35, in commit
get_connection(using).commit()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
self._commit()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/dummy/base.py", line 20, in complain
raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
Copy code
emmmmmm??, Look at the last error message , Say I Django Your database is not configured correctly ? I curiously checked my database configuration and wrote several ORM test , No problems found , So I decided to follow up the error reporting process
2. Problem location
From the error message above , We can see that it mainly starts from the class startup of scheduled tasks
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 271, in __init__
self.setup_schedule()
Copy code
To get django The database connection starts to report an error
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/transaction.py", line 35, in commit
get_connection(using).commit()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
self._commit()
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/dummy/base.py", line 20, in complain
raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
Copy code
See... When getting the link using, I suddenly realized my Django Multiple databases for project configuration , and Django Provided using Parameter is to specify the database currently used , If you don't specify , Route through the configured database or read the default database configuration
Then look up , Before getting the link
File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 261, in schedule_changed
transaction.commit()
Copy code
django-celery-beat
Made a submission , But no database is specified , When getting the database link , Won't you get the corresponding database link through database routing ?
With curiosity, I checked get_connection(using).commit()
Source code
def commit(using=None):
"""Commit a transaction."""
get_connection(using).commit()
Copy code
Further inspection get_connection
Method source code
DEFAULT_DB_ALIAS = 'default'
...
def get_connection(using=None):
""" Get a database connection by name, or the default database connection if no name is provided. This is a private API. """
if using is None:
using = DEFAULT_DB_ALIAS
return connections[using]
Copy code
So far, the case has been solved ,django-celery-beat
There is no need to using Parameters , Lead to Django Will go to get the default database link , and Django When configuring multiple databases , Default database configuration default
It's empty , therefore django-celery-beat
In fact, it does not support multiple databases
3. Solution
After configuring multiple databases , Generally, we need to define the database route ( Otherwise, you will still get default
To configure ),Django After the route is obtained, it will be displayed in the following steps ORM During the operation, we will automatically use the route we defined to match the database corresponding to the current operation
therefore , Here we need to get django-celery-beat
Which database corresponds to the database route we define
Try to modify the source code
Let's try to modify the error in the source code first
""" django_celery_beat/schedulers.py """
from django.db import router, DEFAULT_DB_ALIAS
...... # Other source code
class DatabaseScheduler(Scheduler):
......
# Add the following
@property
def target_db(self):
"""Determine if there is a django route"""
# First check whether it is configured django Database routing
if not settings.DATABASE_ROUTERS:
return DEFAULT_DB_ALIAS
# If the project does not actually implement this method, DEFAULT_DB_ALIAS will be automatically returned.
# The exception will be located to the django routing section
# If there is a route , By calling the of the route db_for_write Method to obtain the corresponding database
# self.Model yes django-celery-beat Model of
db = router.db_for_write(self.Model)
return db
def schedule_changed(self):
try:
......
try:
transaction.commit(using=self.target_db)
except transaction.TransactionManagementError:
pass # not in transaction management.
except DatabaseError as exc:
......
Copy code
Be careful ️:
- If your database routing is not implemented
db_for_write
Method ,Django Will returnDATABASES
indefault
Configuration of , thereforedb_for_write
Must be realized , And must be able todjango-celery-beat
Match the model to the correct database- Remember in
INSTALLED_APPS
Importdjango-celery-beat
Perform migration at the same time
Start again celery-beat
* celery -A skill_test.app beat -l INFO 1 ↵ [email protected]
celery beat v5.1.2 (sun-harmonics) is starting.
__ - ... __ - _
LocalTime -> 2021-11-03 15:24:59
Configuration ->
. broker -> redis://localhost:6379//
. loader -> celery.loaders.app.AppLoader
. scheduler -> django_celery_beat.schedulers.DatabaseScheduler
. logfile -> [stderr]@%INFO
. maxinterval -> 5.00 seconds (5s)
[2021-11-03 15:24:59,065: INFO/MainProcess] beat: Starting...
[2021-11-03 15:24:59,234: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2021-11-03 15:25:00,014: INFO/MainProcess] Scheduler: Sending due task xxx (apps.xxx.tasks.xxx)
Copy code
Successful launch , But we generally do not recommend changing the source code , In the use of celery-beat when , Remember to configure CELERY_BEAT_SCHEDULER Do you , We can support database routing through overloading
Overload mode enables support for Django Multi database configuration
Default configuration is
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
Copy code
Reload this DatabaseScheduler
from django_celery_beat.schedulers import DatabaseScheduler
from django.db import transaction, router
from functools import partial
class SQDatabaseSchedule(DatabaseScheduler):
def sync(self):
with transaction.atomic(using=router.db_for_write(self.Model)):
super(SQDatabaseSchedule, self).sync()
def schedule_changed(self):
db = router.db_for_write(self.Model)
transaction.commit = partial(transaction.commit, using=db)
super(SQDatabaseSchedule, self).schedule_changed()
Copy code
Modify the configuration
CELERY_BEAT_SCHEDULER = 'you_project.you_project_path:DatabaseScheduler'
Copy code
Try starting again
* celery -A skill_test.app beat -l INFO 1 ↵ [email protected]
celery beat v5.1.2 (sun-harmonics) is starting.
__ - ... __ - _
LocalTime -> 2021-11-03 15:26:59
Configuration ->
. broker -> redis://localhost:6379//
. loader -> celery.loaders.app.AppLoader
. scheduler -> django_celery_beat.schedulers.DatabaseScheduler
. logfile -> [stderr]@%INFO
. maxinterval -> 5.00 seconds (5s)
[2021-11-03 15:26:59,065: INFO/MainProcess] beat: Starting...
[2021-11-03 15:26:59,234: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2021-11-03 15:26:00,014: INFO/MainProcess] Scheduler: Sending due task xxx (apps.xxx.tasks.xxx)
Copy code
Successful launch , Be accomplished
copyright notice
author[Shanxiao I],Please bring the original link to reprint, thank you.
https://en.pythonmana.com/2022/01/202201310013176001.html
The sidebar is recommended
- Python notes (20): built in high-order functions
- Python notes (17): closure
- Python notes (18): decorator
- Python notes (16): generators and iterators
- Python notes (XV): List derivation
- Python tells you what timing attacks are
- Python -- file and exception
- [Python from introduction to mastery] (IV) what are the built-in data types of Python? Figure out
- Python code to scan code to pay attention to official account login
- [algorithm learning] 1221 Split balanced string (Java / C / C + + / Python / go / trust)
guess what you like
-
Python notes (22): errors and exceptions
-
Python has been hidden for ten years, and once image recognition is heard all over the world
-
Python notes (21): random number module
-
Python notes (19): anonymous functions
-
Use Python and OpenCV to calculate and draw two-dimensional histogram
-
Python, Hough circle transformation in opencv
-
A library for reading and writing markdown in Python: mdutils
-
Datetime of Python time operation (Part I)
-
The most useful decorator in the python standard library
-
Python iterators and generators
Random recommended
- [Python from introduction to mastery] (V) Python's built-in data types - sequences and strings. They have no girlfriend, not a nanny, and can only be used as dry goods
- Does Python have a, = operator?
- Go through the string common sense in Python
- Fanwai 4 Handling of mouse events and solutions to common problems in Python opencv
- Summary of common functions for processing strings in Python
- When writing Python scripts, be sure to add this
- Python web crawler - Fundamentals (1)
- Pandas handles duplicate values
- Python notes (23): regular module
- Python crawlers are slow? Concurrent programming to understand it
- Parameter passing of Python function
- Stroke tuple in Python
- Talk about ordinary functions and higher-order functions in Python
- [Python data acquisition] page image crawling and saving
- [Python data collection] selenium automated test framework
- Talk about function passing and other supplements in Python
- Python programming simulation poker game
- leetcode 160. Intersection of Two Linked Lists (python)
- Python crawler actual combat, requests module, python to grab the beautiful wallpaper of a station
- Fanwai 5 Detailed description of slider in Python opencv and solutions to common problems
- My friend's stock suffered a terrible loss. When I was angry, I crawled the latest data of securities with Python
- Python interface automation testing framework -- if you want to do well, you must first sharpen its tools
- Python multi thread crawling weather website pictures and saving
- How to convert pandas data to excel file
- Python series tutorials 122
- Python Complete Guide - printing data using pyspark
- Python Complete Guide -- tuple conversion array
- Stroke the list in python (top)
- Analysis of Python requests module
- Comments and variables in Python
- New statement match, the new version of Python is finally going to introduce switch case?
- Fanwai 6 Different operations for image details in Python opencv
- Python crawler native code learning (I)
- Python quantitative data warehouse building series 2: Python operation database
- Python code reading (Part 50): taking elements from list intervals
- Pyechart + pandas made word cloud pictures of 19 report documents
- [Python crawler] multithreaded daemon & join() blocking
- Python crawls cat pictures in batches to realize thousand image imaging
- Van * Python | simple crawling of a planet
- Input and output of Python practice