Solve the problem that Django celery beat prompts that the database is incorrectly configured and does not support multiple databases

1. Problem recurrence

Write a... One day celery When the task is scheduled , start-up celery The rear is ready to start celery-beat, Suddenly found that a large number of warnings pop up and quit the process , Looking up, I found an error

celery beat v5.1.2 (sun-harmonics) is starting.
[2021-11-01 18:32:11,917: INFO/MainProcess] beat: Starting...
[2021-11-01 18:32:12,071: CRITICAL/MainProcess] beat raised exception <class 'django.core.exceptions.ImproperlyConfigured'>: ImproperlyConfigured('settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.')
Traceback (most recent call last):
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/apps/beat.py", line 105, in start_scheduler
    service.start()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 636, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/kombu/utils/objects.py", line 29, in __get__
    return super().__get__(instance, owner)
  File "/Users/systemime/.pyenv/versions/3.9.6/lib/python3.9/functools.py", line 969, in __get__
    val = self.func(instance)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 680, in scheduler
    return self.get_scheduler()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 671, in get_scheduler
    return symbol_by_name(self.scheduler_cls, aliases=aliases)(
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 231, in __init__
    Scheduler.__init__(self, *args, **kwargs)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 271, in __init__
    self.setup_schedule()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 239, in setup_schedule
    self.install_default_entries(self.schedule)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 340, in install_default_entries
    self.update_from_dict(entries)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 328, in update_from_dict
    self.schedule.update(s)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 355, in schedule
    elif self.schedule_changed():
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 261, in schedule_changed
    transaction.commit()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/transaction.py", line 35, in commit
    get_connection(using).commit()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/dummy/base.py", line 20, in complain
    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
 Copy code 

emmmmmm??, Look at the last error message , Say I Django Your database is not configured correctly ？ I curiously checked my database configuration and wrote several ORM test , No problems found , So I decided to follow up the error reporting process

2. Problem location

From the error message above , We can see that it mainly starts from the class startup of scheduled tasks

File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/celery/beat.py", line 271, in __init__
    self.setup_schedule()
 Copy code 

To get django The database connection starts to report an error

File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/transaction.py", line 35, in commit
    get_connection(using).commit()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django/db/backends/dummy/base.py", line 20, in complain
    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
 Copy code 

See... When getting the link using, I suddenly realized my Django Multiple databases for project configuration , and Django Provided using Parameter is to specify the database currently used , If you don't specify , Route through the configured database or read the default database configuration

Then look up , Before getting the link

File "/Users/systemime/.pyenv/versions/3.9.6/envs/vps-django/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 261, in schedule_changed
    transaction.commit()
 Copy code 

django-celery-beat Made a submission , But no database is specified , When getting the database link , Won't you get the corresponding database link through database routing ？

With curiosity, I checked get_connection(using).commit() Source code

def commit(using=None):
    """Commit a transaction."""
    get_connection(using).commit()
 Copy code 

Further inspection get_connection Method source code

DEFAULT_DB_ALIAS = 'default'

...

def get_connection(using=None):
    """ Get a database connection by name, or the default database connection if no name is provided. This is a private API. """
    if using is None:
        using = DEFAULT_DB_ALIAS
    return connections[using]
 Copy code 

So far, the case has been solved ,django-celery-beat There is no need to using Parameters , Lead to Django Will go to get the default database link , and Django When configuring multiple databases , Default database configuration default It's empty , therefore django-celery-beat In fact, it does not support multiple databases

3. Solution

After configuring multiple databases , Generally, we need to define the database route （ Otherwise, you will still get default To configure ）,Django After the route is obtained, it will be displayed in the following steps ORM During the operation, we will automatically use the route we defined to match the database corresponding to the current operation

therefore , Here we need to get django-celery-beat Which database corresponds to the database route we define

Try to modify the source code

Let's try to modify the error in the source code first

""" django_celery_beat/schedulers.py """
from django.db import router, DEFAULT_DB_ALIAS
......  #  Other source code 

class DatabaseScheduler(Scheduler):
    ......

    #  Add the following 
 @property
    def target_db(self):
        """Determine if there is a django route"""
        #  First check whether it is configured django  Database routing 
        if not settings.DATABASE_ROUTERS:
            return DEFAULT_DB_ALIAS
        # If the project does not actually implement this method, DEFAULT_DB_ALIAS will be automatically returned.
        # The exception will be located to the django routing section
        #  If there is a route , By calling the of the route db_for_write Method to obtain the corresponding database 
        # self.Model yes django-celery-beat Model of 
        db = router.db_for_write(self.Model)
        return db
  
    def schedule_changed(self):
        try:
            ......
            try:
                transaction.commit(using=self.target_db)
            except transaction.TransactionManagementError:
                pass  # not in transaction management.
        except DatabaseError as exc:
            ......
  

 Copy code 

Be careful ️：

• If your database routing is not implemented db_for_write Method ,Django Will return DATABASES in default Configuration of , therefore db_for_write Must be realized , And must be able to django-celery-beat Match the model to the correct database
• Remember in INSTALLED_APPS Import django-celery-beat Perform migration at the same time

Start again celery-beat

* celery -A skill_test.app beat -l INFO                                                                                                                                                             1 ↵ [email protected]
celery beat v5.1.2 (sun-harmonics) is starting.
__    -    ... __   -        _
LocalTime -> 2021-11-03 15:24:59
Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
[2021-11-03 15:24:59,065: INFO/MainProcess] beat: Starting...
[2021-11-03 15:24:59,234: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2021-11-03 15:25:00,014: INFO/MainProcess] Scheduler: Sending due task xxx (apps.xxx.tasks.xxx)
 Copy code 

Successful launch , But we generally do not recommend changing the source code , In the use of celery-beat when , Remember to configure CELERY_BEAT_SCHEDULER Do you , We can support database routing through overloading

Overload mode enables support for Django Multi database configuration

Default configuration is

CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
 Copy code 

Reload this DatabaseScheduler

from django_celery_beat.schedulers import DatabaseScheduler
from django.db import transaction, router
from functools import partial


class SQDatabaseSchedule(DatabaseScheduler):

    def sync(self):
        with transaction.atomic(using=router.db_for_write(self.Model)):
            super(SQDatabaseSchedule, self).sync()

    def schedule_changed(self):
        db = router.db_for_write(self.Model)
        transaction.commit = partial(transaction.commit, using=db)
        super(SQDatabaseSchedule, self).schedule_changed()
 Copy code 

Modify the configuration

CELERY_BEAT_SCHEDULER = 'you_project.you_project_path:DatabaseScheduler'
 Copy code 

Try starting again

* celery -A skill_test.app beat -l INFO                                                                                                                                                             1 ↵ [email protected]
celery beat v5.1.2 (sun-harmonics) is starting.
__    -    ... __   -        _
LocalTime -> 2021-11-03 15:26:59
Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
[2021-11-03 15:26:59,065: INFO/MainProcess] beat: Starting...
[2021-11-03 15:26:59,234: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2021-11-03 15:26:00,014: INFO/MainProcess] Scheduler: Sending due task xxx (apps.xxx.tasks.xxx)
 Copy code 

Successful launch , Be accomplished