You can schedule any task with various options with celery: https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown
On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote: > I also have to be able to schedule tasks, so I actually include the > APScheduler package in my projects to make this scheduling. > Can pyramid-tasks be used also in this context? > > > Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss < > pylons-...@googlegroups.com> a écrit : > >> pyramid_tasks looks very promising! thanks for sharing this. i may use >> it in the future. i'll have to see if it is compatible with our setup. >> >> our celery app is mostly tasks that have been migrated from a twisted >> daemon. we already ran web spiders and social-media-account importers >> through twisted, so all the resizing stuff was originally pyramid creating >> database entries (messages) for twisted to process. although we still use >> twisted for those tasks (and more!) migrating to celery for these simpler >> functions has been really productive and timesaving. the code is cleaner >> and easier to maintain, and troubleshooting/bugfixing is so much simpler. >> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com wrote: >> >>> Seems like a good time to plug my new project: >>> https://pypi.org/project/pyramid-tasks/ >>> >>> Pyramid and Celery are so dramatically different it’s a pain to >>> integrate them, and any code shared between them can’t access anything >>> Pyramid-specific. Got tired of trying to make a square peg fit in a round >>> hole, so I brought my entire Pyramid application over to my Celery worker. >>> Having a request object available in a Celery task is incredibly >>> convenient, and my code is much tidier not having to tiptoe around the >>> differences in environments. >>> >>> On Jan 6, 2021, at 2:14 AM, Adam Groszer <agro...@gmail.com> wrote: >>> >>> Hi, >>> >>> Yeah celery is the way to go. >>> There's https://github.com/sontek/pyramid_celery to check out. >>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB >>> with filestorage is not enough, because more processes will need to have >>> access. >>> Do not store valuable data in the celery queue, consider it ephemeral, >>> go through a real DB. >>> >>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote: >>> >>>> Hi Jonathan, >>>> >>>> Thank you for this description of Celery! >>>> I'll try to have a look at it if I can get a little time, it seems to >>>> be a good replacement (probably more reliable!) of my own developments... >>>> >>>> Best regards, >>>> Thierry >>>> -- >>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>> >>>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < >>>> pylons-...@googlegroups.com> a écrit : >>>> >>>>> Thierry, >>>>> >>>>> That is what I mostly use Celery for (it's also used for generating >>>>> Reports, ACME SSL Certificates and a few other things). A Pyramid Request >>>>> will defer a job(s) to Celery via the "Transport Backend" and receive a >>>>> task/messaging ID. Subsequent requests will poll the Celery "Result >>>>> Backend" for the status of that job, based on the ID. This way we show >>>>> the "Still processing!" message to users. >>>>> >>>>> Celery is run as a worker with multiple processes. The Celery task >>>>> manager grabs a task off the queue, then does the resizing, uploads to >>>>> S3, >>>>> and notifies the Result Backend when complete. I use Redis for Result >>>>> and >>>>> Transport. I think there was once a ZeroMQ integration; there is >>>>> definitely RabbitMQ integration, it's one of the more popular options. >>>>> >>>>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com >>>>> wrote: >>>>> >>>>>> Hi Jonathan, >>>>>> >>>>>> I didn't have a look at Celery yet, maybe it could work... >>>>>> My goal is to be able to start long-running tasks (medias files >>>>>> conversions for example) whose definition is stored in database and >>>>>> managed >>>>>> by end users from my main Pyramid application; for this, I create a >>>>>> dedicated sub-process which is "managed" using ZeroMQ messages (I use >>>>>> this >>>>>> because it's also used for other non-Python/Pyramid applications). >>>>>> Tasks are then started using dedicated threads. >>>>>> >>>>>> Everything is OK until now... My only requirement now is to be able >>>>>> to get access to main Pyramid's registry from my sub-process and it's >>>>>> threads, and I don't really understand why I get a pointer to >>>>>> ZCA global registry... :( >>>>>> >>>>>> Best regards, >>>>>> Thierry >>>>>> -- >>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>> >>>>>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >>>>>> pylons-...@googlegroups.com> a écrit : >>>>>> >>>>>>> Have you considered using Celery for this? I started offloading >>>>>>> everything related to subprocesses and message queues to it a while >>>>>>> back, >>>>>>> and have been much happier. >>>>>>> >>>>>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I need to create custom sub-processes from my main Pyramid >>>>>>>> application; these processes are used to handle "commands" received >>>>>>>> from >>>>>>>> ZeroMQ messages. >>>>>>>> If I use the components registry to register adapters and utilities >>>>>>>> which are also required in these processes, is there a way to start >>>>>>>> these >>>>>>>> processes so that they can "inherit" from the main application >>>>>>>> registry ? >>>>>>>> >>>>>>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>>>>>> "config.hook_zca()" on application startup before doing >>>>>>>> the "config.setup_registry()" and "config.scan()". >>>>>>>> I tried to do the same operations in my sub-process "run" method, >>>>>>>> but I always get the "global" registry instead of my "webapp" registry >>>>>>>> which was configured in my main application... >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Thierry >>>>>>>> -- >>>>>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "pylons-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to pylons-discus...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "pylons-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to pylons-discus...@googlegroups.com. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pylons-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pylons-discus...@googlegroups.com. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/627ab125-7fb3-429d-ab43-7340caf75b5en%40googlegroups.com.
