Seems like a good time to plug my new project: https://pypi.org/project/pyramid-tasks/ <https://pypi.org/project/pyramid-tasks/>
Pyramid and Celery are so dramatically different it’s a pain to integrate them, and any code shared between them can’t access anything Pyramid-specific. Got tired of trying to make a square peg fit in a round hole, so I brought my entire Pyramid application over to my Celery worker. Having a request object available in a Celery task is incredibly convenient, and my code is much tidier not having to tiptoe around the differences in environments. > On Jan 6, 2021, at 2:14 AM, Adam Groszer <agros...@gmail.com> wrote: > > Hi, > > Yeah celery is the way to go. > There's https://github.com/sontek/pyramid_celery to check out. > Some things to keep in mind is, you'll need a shared DB, good-old ZODB with > filestorage is not enough, because more processes will need to have access. > Do not store valuable data in the celery queue, consider it ephemeral, go > through a real DB. > > On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote: > Hi Jonathan, > > Thank you for this description of Celery! > I'll try to have a look at it if I can get a little time, it seems to be a > good replacement (probably more reliable!) of my own developments... > > Best regards, > Thierry > -- > https://www.ulthar.net <https://www.ulthar.net/> -- > http://pyams.readthedocs.io <http://pyams.readthedocs.io/> > Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss > <pylons-...@googlegroups.com > <applewebdata://05B86B91-CBEF-4B2D-80B4-7758B394E393>> a écrit : > Thierry, > > That is what I mostly use Celery for (it's also used for generating Reports, > ACME SSL Certificates and a few other things). A Pyramid Request will defer a > job(s) to Celery via the "Transport Backend" and receive a task/messaging ID. > Subsequent requests will poll the Celery "Result Backend" for the status of > that job, based on the ID. This way we show the "Still processing!" message > to users. > > Celery is run as a worker with multiple processes. The Celery task manager > grabs a task off the queue, then does the resizing, uploads to S3, and > notifies the Result Backend when complete. I use Redis for Result and > Transport. I think there was once a ZeroMQ integration; there is definitely > RabbitMQ integration, it's one of the more popular options. > > On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com > <applewebdata://05B86B91-CBEF-4B2D-80B4-7758B394E393> wrote: > Hi Jonathan, > > I didn't have a look at Celery yet, maybe it could work... > My goal is to be able to start long-running tasks (medias files conversions > for example) whose definition is stored in database and managed by end users > from my main Pyramid application; for this, I create a dedicated sub-process > which is "managed" using ZeroMQ messages (I use this because it's also used > for other non-Python/Pyramid applications). > Tasks are then started using dedicated threads. > > Everything is OK until now... My only requirement now is to be able to get > access to main Pyramid's registry from my sub-process and it's threads, and I > don't really understand why I get a pointer to > ZCA global registry... :( > > Best regards, > Thierry > -- > https://www.ulthar.net <https://www.ulthar.net/> -- > http://pyams.readthedocs.io <http://pyams.readthedocs.io/> > Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss > <pylons-...@googlegroups.com <>> a écrit : > Have you considered using Celery for this? I started offloading everything > related to subprocesses and message queues to it a while back, and have been > much happier. > > On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com <> wrote: > Hi, > > I need to create custom sub-processes from my main Pyramid application; these > processes are used to handle "commands" received from ZeroMQ messages. > If I use the components registry to register adapters and utilities which are > also required in these processes, is there a way to start these processes so > that they can "inherit" from the main application registry ? > > I'm actually using Pyramid 1.10.5, and I generally do a "config.hook_zca()" > on application startup before doing the "config.setup_registry()" and > "config.scan()". > I tried to do the same operations in my sub-process "run" method, but I > always get the "global" registry instead of my "webapp" registry which was > configured in my main application... > > Best regards, > Thierry > -- > https://www.ulthar.net <https://www.ulthar.net/> -- > http://pyams.readthedocs.io <http://pyams.readthedocs.io/> > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discus...@googlegroups.com <>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com > > <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discus...@googlegroups.com > <applewebdata://05B86B91-CBEF-4B2D-80B4-7758B394E393>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com > > <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com > <mailto:pylons-discuss+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com > > <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5182A396-074C-4374-A9F3-797E6B81FAEC%40luhn.com.
