Thanks for your work.
Unfortunately i can not test this change in a timely manner as it
depends on too many changes compared to stable 6.2 which we run.
/Valentin
On 09/11/2020 17.56, Mads Kiilerich wrote:
On 11/5/20 1:11 AM, Mads Kiilerich wrote:
On 11/3/20 5:08 PM, Valentin Kleibel wrote:
Hi,
We are using kallithea with a postgres database on another server
connected via ssl.
It seems that celery in version 4 uses prefork in its worker pool
model now, which leads to issues with such a database connection:
I think you are right something is wrong.
First, the celery-run will not only take a config file as parameter
and read it. It will also initialize the WSGI app and database
connection, before calling into celery. Effectively making it
pre-fork. I think you are right that we only should initialize things
after celery has forked and started worker processes.
The first changes on
https://kallithea-scm.org/repos/kallithea/pull-request/292/_/celery_and_stuff
do this. After some refactorings (that may or may not be essential for
the change), I think the proper fix is in
https://kallithea-scm.org/repos/kallithea-incoming/changeset/e32c37b84c53bc6342e7b4698cb06168a7bcff15
.
I don't think we should have to dispose the engine. We just shouldn't
initialize it pre-fork.
Does this work for you?
Disposing the engine for each task seems inefficient and shouldn't be
necessary. As an alternative to your patch for disposing in dbsession,
it could perhaps just dispose in
kallithea.CELERY_APP.loader.on_worker_process_init .
But also, it might be a bit odd that we have the celery-run entry
point. It would perhaps be better to somehow use "celery" as entry
point and let it call into Kallithea in each worker. But I haven't
investigated how feasible that would be.
This is still TODO.
There is still a lot of room for improvement, making sure that we use
Celery in a simple and recommended way, avoiding the need for
initializing Celery at *import* time, and upgrading to Celery 5.
Contributions in this area would be welcome ;-)
/Mads
_______________________________________________
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
