I’m not familiar with APScheduler. celery-beat seems to be the de facto way to run scheduled tasks in Celery. https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html <https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html>
celery-beat should be compatible with pyramid-tasks, but I haven’t tested or documented it yet. I personally use AWS CloudWatch Events -> SNS -> HTTP to my app -> validate_aws_sns_message -> trigger Celery task. > On Jan 6, 2021, at 11:22 AM, Thierry Florac <tflo...@gmail.com> wrote: > > I also have to be able to schedule tasks, so I actually include the > APScheduler package in my projects to make this scheduling. > Can pyramid-tasks be used also in this context? > > > Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss > <pylons-discuss@googlegroups.com <mailto:pylons-discuss@googlegroups.com>> a > écrit : > pyramid_tasks looks very promising! thanks for sharing this. i may use it in > the future. i'll have to see if it is compatible with our setup. > > our celery app is mostly tasks that have been migrated from a twisted daemon. > we already ran web spiders and social-media-account importers through > twisted, so all the resizing stuff was originally pyramid creating database > entries (messages) for twisted to process. although we still use twisted for > those tasks (and more!) migrating to celery for these simpler functions has > been really productive and timesaving. the code is cleaner and easier to > maintain, and troubleshooting/bugfixing is so much simpler. > On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com > <mailto:the...@luhn.com> wrote: > Seems like a good time to plug my new project: > https://pypi.org/project/pyramid-tasks/ > <https://pypi.org/project/pyramid-tasks/> > > Pyramid and Celery are so dramatically different it’s a pain to integrate > them, and any code shared between them can’t access anything > Pyramid-specific. Got tired of trying to make a square peg fit in a round > hole, so I brought my entire Pyramid application over to my Celery worker. > Having a request object available in a Celery task is incredibly convenient, > and my code is much tidier not having to tiptoe around the differences in > environments. > > >> On Jan 6, 2021, at 2:14 AM, Adam Groszer <agro...@gmail.com <>> wrote: >> > >> Hi, >> >> Yeah celery is the way to go. >> There's https://github.com/sontek/pyramid_celery >> <https://github.com/sontek/pyramid_celery> to check out. >> Some things to keep in mind is, you'll need a shared DB, good-old ZODB with >> filestorage is not enough, because more processes will need to have access. >> Do not store valuable data in the celery queue, consider it ephemeral, go >> through a real DB. >> >> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com >> <http://gmail.com/> wrote: >> Hi Jonathan, >> >> Thank you for this description of Celery! >> I'll try to have a look at it if I can get a little time, it seems to be a >> good replacement (probably more reliable!) of my own developments... >> >> Best regards, >> Thierry >> -- >> https://www.ulthar.net <https://www.ulthar.net/> -- >> http://pyams.readthedocs.io <http://pyams.readthedocs.io/> >> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss >> <pylons-...@googlegroups.com <>> a écrit : >> Thierry, >> >> That is what I mostly use Celery for (it's also used for generating Reports, >> ACME SSL Certificates and a few other things). A Pyramid Request will defer >> a job(s) to Celery via the "Transport Backend" and receive a task/messaging >> ID. Subsequent requests will poll the Celery "Result Backend" for the >> status of that job, based on the ID. This way we show the "Still >> processing!" message to users. >> >> Celery is run as a worker with multiple processes. The Celery task manager >> grabs a task off the queue, then does the resizing, uploads to S3, and >> notifies the Result Backend when complete. I use Redis for Result and >> Transport. I think there was once a ZeroMQ integration; there is definitely >> RabbitMQ integration, it's one of the more popular options. >> >> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com <> wrote: >> Hi Jonathan, >> >> I didn't have a look at Celery yet, maybe it could work... >> My goal is to be able to start long-running tasks (medias files conversions >> for example) whose definition is stored in database and managed by end users >> from my main Pyramid application; for this, I create a dedicated sub-process >> which is "managed" using ZeroMQ messages (I use this because it's also used >> for other non-Python/Pyramid applications). >> Tasks are then started using dedicated threads. >> >> Everything is OK until now... My only requirement now is to be able to get >> access to main Pyramid's registry from my sub-process and it's threads, and >> I don't really understand why I get a pointer to >> ZCA global registry... :( >> >> Best regards, >> Thierry >> -- >> https://www.ulthar.net <https://www.ulthar.net/> -- >> http://pyams.readthedocs.io <http://pyams.readthedocs.io/> >> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss >> <pylons-...@googlegroups.com <>> a écrit : >> Have you considered using Celery for this? I started offloading everything >> related to subprocesses and message queues to it a while back, and have been >> much happier. >> >> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com <> wrote: >> Hi, >> >> I need to create custom sub-processes from my main Pyramid application; >> these processes are used to handle "commands" received from ZeroMQ messages. >> If I use the components registry to register adapters and utilities which >> are also required in these processes, is there a way to start these >> processes so that they can "inherit" from the main application registry ? >> >> I'm actually using Pyramid 1.10.5, and I generally do a "config.hook_zca()" >> on application startup before doing the "config.setup_registry()" and >> "config.scan()". >> I tried to do the same operations in my sub-process "run" method, but I >> always get the "global" registry instead of my "webapp" registry which was >> configured in my main application... >> >> Best regards, >> Thierry >> -- >> https://www.ulthar.net <https://www.ulthar.net/> -- >> http://pyams.readthedocs.io <http://pyams.readthedocs.io/> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com <>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com <>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com <>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com > <mailto:pylons-discuss+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com > > <https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pylons-discuss+unsubscr...@googlegroups.com > <mailto:pylons-discuss+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC5EhcCj%2BRmr_ABMObbgYtTUYskaSR8TL0Nexm_1fCZ-A%40mail.gmail.com > > <https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC5EhcCj%2BRmr_ABMObbgYtTUYskaSR8TL0Nexm_1fCZ-A%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/7145F71E-2490-4D75-93DD-3CE9CCCF94DB%40luhn.com.
