pyramid_tasks looks very promising! thanks for sharing this. i may use it in the future. i'll have to see if it is compatible with our setup.
our celery app is mostly tasks that have been migrated from a twisted daemon. we already ran web spiders and social-media-account importers through twisted, so all the resizing stuff was originally pyramid creating database entries (messages) for twisted to process. although we still use twisted for those tasks (and more!) migrating to celery for these simpler functions has been really productive and timesaving. the code is cleaner and easier to maintain, and troubleshooting/bugfixing is so much simpler. On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote: > Seems like a good time to plug my new project: > https://pypi.org/project/pyramid-tasks/ > > Pyramid and Celery are so dramatically different it’s a pain to integrate > them, and any code shared between them can’t access anything > Pyramid-specific. Got tired of trying to make a square peg fit in a round > hole, so I brought my entire Pyramid application over to my Celery worker. > Having a request object available in a Celery task is incredibly > convenient, and my code is much tidier not having to tiptoe around the > differences in environments. > > On Jan 6, 2021, at 2:14 AM, Adam Groszer <agro...@gmail.com> wrote: > > Hi, > > Yeah celery is the way to go. > There's https://github.com/sontek/pyramid_celery to check out. > Some things to keep in mind is, you'll need a shared DB, good-old ZODB > with filestorage is not enough, because more processes will need to have > access. > Do not store valuable data in the celery queue, consider it ephemeral, go > through a real DB. > > On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote: > >> Hi Jonathan, >> >> Thank you for this description of Celery! >> I'll try to have a look at it if I can get a little time, it seems to be >> a good replacement (probably more reliable!) of my own developments... >> >> Best regards, >> Thierry >> -- >> https://www.ulthar.net -- http://pyams.readthedocs.io >> >> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < >> pylons-...@googlegroups.com> a écrit : >> >>> Thierry, >>> >>> That is what I mostly use Celery for (it's also used for generating >>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request >>> will defer a job(s) to Celery via the "Transport Backend" and receive a >>> task/messaging ID. Subsequent requests will poll the Celery "Result >>> Backend" for the status of that job, based on the ID. This way we show >>> the "Still processing!" message to users. >>> >>> Celery is run as a worker with multiple processes. The Celery task >>> manager grabs a task off the queue, then does the resizing, uploads to S3, >>> and notifies the Result Backend when complete. I use Redis for Result and >>> Transport. I think there was once a ZeroMQ integration; there is >>> definitely RabbitMQ integration, it's one of the more popular options. >>> >>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote: >>> >>>> Hi Jonathan, >>>> >>>> I didn't have a look at Celery yet, maybe it could work... >>>> My goal is to be able to start long-running tasks (medias files >>>> conversions for example) whose definition is stored in database and >>>> managed >>>> by end users from my main Pyramid application; for this, I create a >>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use this >>>> because it's also used for other non-Python/Pyramid applications). >>>> Tasks are then started using dedicated threads. >>>> >>>> Everything is OK until now... My only requirement now is to be able to >>>> get access to main Pyramid's registry from my sub-process and it's >>>> threads, >>>> and I don't really understand why I get a pointer to >>>> ZCA global registry... :( >>>> >>>> Best regards, >>>> Thierry >>>> -- >>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>> >>>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >>>> pylons-...@googlegroups.com> a écrit : >>>> >>>>> Have you considered using Celery for this? I started offloading >>>>> everything related to subprocesses and message queues to it a while back, >>>>> and have been much happier. >>>>> >>>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I need to create custom sub-processes from my main Pyramid >>>>>> application; these processes are used to handle "commands" received from >>>>>> ZeroMQ messages. >>>>>> If I use the components registry to register adapters and utilities >>>>>> which are also required in these processes, is there a way to start >>>>>> these >>>>>> processes so that they can "inherit" from the main application registry ? >>>>>> >>>>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>>>> "config.hook_zca()" on application startup before doing >>>>>> the "config.setup_registry()" and "config.scan()". >>>>>> I tried to do the same operations in my sub-process "run" method, but >>>>>> I always get the "global" registry instead of my "webapp" registry which >>>>>> was configured in my main application... >>>>>> >>>>>> Best regards, >>>>>> Thierry >>>>>> -- >>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "pylons-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to pylons-discus...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pylons-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pylons-discus...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discus...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com > > <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com.
