Hi Jonathan, Thank you for this description of Celery! I'll try to have a look at it if I can get a little time, it seems to be a good replacement (probably more reliable!) of my own developments...
Best regards, Thierry -- https://www.ulthar.net -- http://pyams.readthedocs.io Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < pylons-discuss@googlegroups.com> a écrit : > Thierry, > > That is what I mostly use Celery for (it's also used for generating > Reports, ACME SSL Certificates and a few other things). A Pyramid Request > will defer a job(s) to Celery via the "Transport Backend" and receive a > task/messaging ID. Subsequent requests will poll the Celery "Result > Backend" for the status of that job, based on the ID. This way we show > the "Still processing!" message to users. > > Celery is run as a worker with multiple processes. The Celery task manager > grabs a task off the queue, then does the resizing, uploads to S3, and > notifies the Result Backend when complete. I use Redis for Result and > Transport. I think there was once a ZeroMQ integration; there is > definitely RabbitMQ integration, it's one of the more popular options. > > On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote: > >> Hi Jonathan, >> >> I didn't have a look at Celery yet, maybe it could work... >> My goal is to be able to start long-running tasks (medias files >> conversions for example) whose definition is stored in database and managed >> by end users from my main Pyramid application; for this, I create a >> dedicated sub-process which is "managed" using ZeroMQ messages (I use this >> because it's also used for other non-Python/Pyramid applications). >> Tasks are then started using dedicated threads. >> >> Everything is OK until now... My only requirement now is to be able to >> get access to main Pyramid's registry from my sub-process and it's threads, >> and I don't really understand why I get a pointer to >> ZCA global registry... :( >> >> Best regards, >> Thierry >> -- >> https://www.ulthar.net -- http://pyams.readthedocs.io >> >> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >> pylons-...@googlegroups.com> a écrit : >> >>> Have you considered using Celery for this? I started offloading >>> everything related to subprocesses and message queues to it a while back, >>> and have been much happier. >>> >>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote: >>> >>>> Hi, >>>> >>>> I need to create custom sub-processes from my main Pyramid application; >>>> these processes are used to handle "commands" received from ZeroMQ >>>> messages. >>>> If I use the components registry to register adapters and utilities >>>> which are also required in these processes, is there a way to start these >>>> processes so that they can "inherit" from the main application registry ? >>>> >>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>> "config.hook_zca()" on application startup before doing >>>> the "config.setup_registry()" and "config.scan()". >>>> I tried to do the same operations in my sub-process "run" method, but I >>>> always get the "global" registry instead of my "webapp" registry which was >>>> configured in my main application... >>>> >>>> Best regards, >>>> Thierry >>>> -- >>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pylons-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pylons-discus...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com > <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC46yjf55pcMEYKmjv3%3DpwBP8%2BvTs%2B_vdn5uBySpLHNsg%40mail.gmail.com.
