I have a question about running Channels and Django code under Celery.
Originally, I had some code that used neither Channels nor Celery, but
simply run as part of the View code:
def import_all(...)
...database access...
This is an expensive operation and to avoid timing out the browser, I
needed to (a) run it under Celery but also (b) provide progress updates via
a Websocket to the client. First, I implemented (b) using a Channels
Consumer and the group_send, and that worked fine. I then tried to
implement (a) using a Celery Task:
@task
def import_all(...):
...database access...
async_to_sync(group_send)(...)
However, this caused Celery to be unhappy and the worker to die:
...
2020-08-22 13:33:08,303 [ERROR] MainProcess: Task handler raised error:
WorkerLostError('Worker exited prematurely: exitcode 0.')
Traceback (most recent call last):
File
"/usr/local/lib/python3.8/dist-packages/billiard-3.6.3.0-py3.8.egg/billiard/pool.py",
line 1265, in mark_as_worker_lost
raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0.
After quite some Googling, and various detours including where the DataBase
access caused issues (it seems it cannot be run from an async context), I
ended up with this double-wrapped monster:
@task
def import_all(...):
async_to_sync(_import_all)(...)
async def _import_all(...):
await database_sync_to_async(__import_all)(...)
def __import_all(...): <<<<<<<<<< original code
...database access...
async_to_sync(group_send)(...)
This seems to work (even though the Celery task eventually dies, it seems
to have done its work first, and is replaced). Is this double-wrapping
really the correct approach? Even if it is clumsy, is there any reason to
think it won't be reliable?
TIA, Shaheed
P.S. I should mention that I am using Celery's prefork library, as
opposed to say gevent or greenlets.
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CAHAc2jdxhaM%3Di6_aLRO-a3qdehkXt1tWYN%2B_oDpBXsB0Hp2jww%40mail.gmail.com.
