Re: [pylons-discuss] Creating sub-processes from Pyramid application

[Theron Luhn]

Hi Thierry,

I’ve added support for periodic tasks in Pyramid Tasks.  
https://github.com/luhn/pyramid-tasks#periodic-tasks

> On Jan 6, 2021, at 11:46 PM, Adam Groszer  wrote:
> 
> You can schedule any task with various options with celery:
> 
> https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown
> 
> On Wednesday, January 6, 2021 at 8:22:28 PM UTC+1 tfl...@gmail.com wrote:
> I also have to be able to schedule tasks, so I actually include the 
> APScheduler package in my projects to make this scheduling.
> Can pyramid-tasks be used also in this context?
> 
> 
> Le mer. 6 janv. 2021 à 19:44, 'Jonathan Vanasco' via pylons-discuss 
>  > a écrit :
> pyramid_tasks looks very promising! thanks for sharing this.  i may use it in 
> the future.  i'll have to see if it is compatible with our setup.
> 
> our celery app is mostly tasks that have been migrated from a twisted daemon. 
>  we already ran web spiders and social-media-account importers through 
> twisted, so all the resizing stuff was originally pyramid creating database 
> entries (messages) for twisted to process.  although we still use twisted for 
> those tasks (and more!) migrating to celery for these simpler functions has 
> been really productive and timesaving.  the code is cleaner and easier to 
> maintain, and troubleshooting/bugfixing is so much simpler.
> On Wednesday, January 6, 2021 at 12:51:37 PM UTC-5 the...@luhn.com 
>  wrote:
> Seems like a good time to plug my new project:  
> https://pypi.org/project/pyramid-tasks/ 
> 
> 
> Pyramid and Celery are so dramatically different it’s a pain to integrate 
> them, and any code shared between them can’t access anything 
> Pyramid-specific.  Got tired of trying to make a square peg fit in a round 
> hole, so I brought my entire Pyramid application over to my Celery worker.  
> Having a request object available in a Celery task is incredibly convenient, 
> and my code is much tidier not having to tiptoe around the differences in 
> environments.
> 
> 
>> On Jan 6, 2021, at 2:14 AM, Adam Groszer > wrote:
>> 
> 
>> Hi,
>> 
>> Yeah celery is the way to go.
>> There's https://github.com/sontek/pyramid_celery 
>>  to check out.
>> Some things to keep in mind is, you'll need a shared DB, good-old ZODB with 
>> filestorage is not enough, because more processes will need to have access.
>> Do not store valuable data in the celery queue, consider it ephemeral, go 
>> through a real DB.
>> 
>> On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com 
>>  wrote:
>> Hi Jonathan,
>> 
>> Thank you for this description of Celery!
>> I'll try to have a look at it if I can get a little time, it seems to be a 
>> good replacement (probably more reliable!) of my own developments...
>> 
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net  -- 
>> http://pyams.readthedocs.io 
>> Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss 
>> > a écrit :
>> Thierry,
>> 
>> That is what I mostly use Celery for (it's also used for generating Reports, 
>> ACME SSL Certificates and a few other things). A Pyramid Request will defer 
>> a job(s) to Celery via the "Transport Backend" and receive a task/messaging 
>> ID.  Subsequent requests will poll the Celery "Result Backend"  for the 
>> status of that job, based on the ID.  This way we show the "Still 
>> processing!" message to users.
>> 
>> Celery is run as a worker with multiple processes. The Celery task manager 
>> grabs a task off the queue, then does the resizing, uploads to S3, and 
>> notifies the Result Backend when complete.  I use Redis for Result and 
>> Transport.  I think there was once a ZeroMQ integration; there is definitely 
>> RabbitMQ integration, it's one of the more popular options.
>> 
>> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com <> wrote:
>> Hi Jonathan,
>> 
>> I didn't have a look at Celery yet, maybe it could work...
>> My goal is to be able to start long-running tasks (medias files conversions 
>> for example) whose definition is stored in database and managed by end users 
>> from my main Pyramid application; for this, I create a dedicated sub-process 
>> which is "managed" using ZeroMQ messages (I use this because it's also used 
>> for other non-Python/Pyramid applications).
>> Tasks are then started using dedicated threads.
>> 
>> Everything is OK until now... My only requirement now is to be able to get 
>> access to main Pyramid's registry from my sub-process and it's threads, and 
>> I don't really understand why I get a pointer to
>> ZCA global registry...  :(
>> 
>> Best regards,
>> Thierry
>> -- 
>>   https://www.ulthar.net  -- 
>> http://pyams.readthedocs.io 
>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss 
>> > a écrit :

Re: [pylons-discuss] Re: Pyramid maintenance mode

[Mike Orr]

Don't forget the 'asbool()'. Without it the 'in_maintenance' text
setting value may be interpreted wrong. E.g., '0' or ' ' or 'false'
would evaluate to True.

On Sat, Jan 9, 2021 at 11:30 AM Mike Orr  wrote:
>
> The includeme function takes care of all the tween registration for
> you, so you just have to include the module using the
> "pyramid.includes" setting or such. You've got that working.
>
> Since you're enabling maintenance mode with a config setting, and
> you'll have to restart the application to change it, you can hoist the
> 'in_maintenance' setting check into the includeme function. E.g.,
>
> if registry.settings.get("in_maintenance", False):
> tween = __name__ + ".maintenance_tween_factory"
> config.add_tween(tween)
>
> That way the tween will be registered only if maintenance mode is
> enabled. That will simplify your tween code slightly and is one
> millisecond more efficient.
>
> As for your other problems of the maintenance screen never appearing,
> or not appearing with the right URLs (I'm not sure which), I would add
> some print or logging statements in the tween function to confirm it's
> being called and see what its request.url, request,matchdict, and
> request.path are. (I prefer request.path over request.url.) They may
> be different from what you expect, and if so we'd need to figure out
> why.
>
> I would also simplify the 'if not ... or not ...' expressions. While I
> don't see anything in particular wrong with them, there may be a logic
> error buried in the expressions somewhere, and separating them out and
> printing the intermediate values may reveal what it is
>
> On Fri, Jan 8, 2021 at 9:06 AM C J  wrote:
> >
> > I try to understand why my "or not" have no effect:
> >
> > def maintenance_tween_factory(handler, registry):
> > # Return a tween callable.
> > # 'handler' is the next tween or the WSGI application.
> > # Deployment settings are in 'registry.settings'.
> > def maintenance_tween(request):
> > #print(request)
> > is_maintenance_mode = registry.settings.get("in_maintenance")
> > if is_maintenance_mode == "True":
> > print(request.url)
> > print("maintenance" in request.url) # True but still affected 
> > by the maintenance mode
> > url = request.route_url('maintenance')
> > if \
> > not "/api/dbmigration" in request.url \
> > or not "maintenance" in request.url:
> > return HTTPServiceUnavailable(
> > "Please note that we will be performing important 
> > server maintenance in a few minutes, during which time the server will be 
> > unavailable. If you are in the middle of something important, please save 
> > your work or hold off on any critical actions until we are finished."
> >  )
> > #return HTTPServiceUnavailable(location=url)
> >
> > # # Could limit it to certain request.path's.
> > # if not ("/api/dbmigration" in request.url and is_maintenance_mode 
> > == "True") \
> > # or not ("maintenance" in request.url and 
> > is_maintenance_mode == "True"):
> > # # Return an error response, bypassing the application.
> > # return HTTPServiceUnavailable(
> > # "Please note that we will be performing important 
> > server maintenance in a few minutes, during which time the server will be 
> > unavailable. If you are in the middle of something important, please save 
> > your work or hold off on any critical actions until we are finished."
> > # )
> > else:
> > # Call next tween/application and return its response 
> > unchanged.
> > return handler(request)
> > else:
> > # Call next tween/application and return its response unchanged.
> > return handler(request)
> > return maintenance_tween
> >>>
> >>>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "pylons-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to pylons-discuss+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/pylons-discuss/3ea70938-5f98-4a6b-97ba-25dea5edc50dn%40googlegroups.com.
>
>
>
> --
> Mike Orr 



-- 
Mike Orr 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurawVFxb5XRdWrq5Qu8GjgrO8qNcGqHpymxGMZfLT3A7w%40mail.gmail.com.

Re: [pylons-discuss] Re: Pyramid maintenance mode

[Mike Orr]

The includeme function takes care of all the tween registration for
you, so you just have to include the module using the
"pyramid.includes" setting or such. You've got that working.

Since you're enabling maintenance mode with a config setting, and
you'll have to restart the application to change it, you can hoist the
'in_maintenance' setting check into the includeme function. E.g.,

if registry.settings.get("in_maintenance", False):
tween = __name__ + ".maintenance_tween_factory"
config.add_tween(tween)

That way the tween will be registered only if maintenance mode is
enabled. That will simplify your tween code slightly and is one
millisecond more efficient.

As for your other problems of the maintenance screen never appearing,
or not appearing with the right URLs (I'm not sure which), I would add
some print or logging statements in the tween function to confirm it's
being called and see what its request.url, request,matchdict, and
request.path are. (I prefer request.path over request.url.) They may
be different from what you expect, and if so we'd need to figure out
why.

I would also simplify the 'if not ... or not ...' expressions. While I
don't see anything in particular wrong with them, there may be a logic
error buried in the expressions somewhere, and separating them out and
printing the intermediate values may reveal what it is

On Fri, Jan 8, 2021 at 9:06 AM C J  wrote:
>
> I try to understand why my "or not" have no effect:
>
> def maintenance_tween_factory(handler, registry):
> # Return a tween callable.
> # 'handler' is the next tween or the WSGI application.
> # Deployment settings are in 'registry.settings'.
> def maintenance_tween(request):
> #print(request)
> is_maintenance_mode = registry.settings.get("in_maintenance")
> if is_maintenance_mode == "True":
> print(request.url)
> print("maintenance" in request.url) # True but still affected by 
> the maintenance mode
> url = request.route_url('maintenance')
> if \
> not "/api/dbmigration" in request.url \
> or not "maintenance" in request.url:
> return HTTPServiceUnavailable(
> "Please note that we will be performing important server 
> maintenance in a few minutes, during which time the server will be 
> unavailable. If you are in the middle of something important, please save 
> your work or hold off on any critical actions until we are finished."
>  )
> #return HTTPServiceUnavailable(location=url)
>
> # # Could limit it to certain request.path's.
> # if not ("/api/dbmigration" in request.url and is_maintenance_mode 
> == "True") \
> # or not ("maintenance" in request.url and 
> is_maintenance_mode == "True"):
> # # Return an error response, bypassing the application.
> # return HTTPServiceUnavailable(
> # "Please note that we will be performing important 
> server maintenance in a few minutes, during which time the server will be 
> unavailable. If you are in the middle of something important, please save 
> your work or hold off on any critical actions until we are finished."
> # )
> else:
> # Call next tween/application and return its response 
> unchanged.
> return handler(request)
> else:
> # Call next tween/application and return its response unchanged.
> return handler(request)
> return maintenance_tween
>>>
>>>
> --
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/3ea70938-5f98-4a6b-97ba-25dea5edc50dn%40googlegroups.com.



-- 
Mike Orr 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Dup5QbwVk%2BR1RixLPEHV%3DqMsA%2BCTUXO6RbGEy-sz6R73xg%40mail.gmail.com.