Re: [pylons-discuss] Confused accessing db sessions

[Jonathan Vanasco]

On Monday, January 6, 2020 at 4:03:19 PM UTC-5, Kate Boelhauf wrote:
>
> This is insanely helpful - thank you Michael.  I'm going to pass the 
> dbsession into the class method for now but look into a service layer.  
>

A few years ago I adopted the pattern of passing Pyramid's `request` object 
into all class methods - and pretty much everything else - as the first 
argument representing the "context".  From that, I can grab my active core 
database session via `request.dbsession`.  If I need to override with an 
explicit dbSession, I pass in a dbsession kwarg or will inspect the object 
using `sqlalchemy.orm.object_session(user)` as Michael suggested.

At one point I passed in the dbSession, but I found myself with a chunks of 
code that expected a dbSession and chunks that expected a request.  
Standardizing everything to a request saved a lot of headaches down the 
line for me.

>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/89768643-b6bc-4458-b0ee-b8b8f3d57b54%40googlegroups.com.

Re: [pylons-discuss] question regarding url dispatch with very similar routes

[Jonathan Vanasco]

On Thursday, December 19, 2019 at 12:26:34 PM UTC-5, Michael Merickel wrote:
>
> You didn't make it completely clear what you're doing with your 
> view_config but I assume it's something to do with the match_param 
> predicate after your factory updates the matchdict. That's fine, and you 
> could use it to annotate the request with another attribute and then write 
> a custom predicate for your views to use instead of the match_param. Think 
> something like request.user_focus = 'a_username', 
> and @view_config(route_name='user_focus', user_type='a_username') which 
> will work well if they are mutually exclusive.
>

Thanks, Michael.  I'm not doing that, but it looks like i should!

Your reply gave me enough insight/hints to get something working, and I'm 
pretty happy with it from a maintainability standpoint.

I'm now doing the following:



def factory__user_type_discriminator(request):
"""inspects the matchdict's 'integrated_user_identifier' placeholder;
   sets 'user_type=A' or 'user_type=B' in the matchdict
"""
pass

config.add_route("integrated_user_focus",
 "/users/{integrated_user_identifier}",
 factory=factory__user_type_discriminator
 )


class ViewClass(handler):

@view_config(route_name="integrated_user_focus", match_param=
'user_type=A')
 def view_type_a(self, request):
pass
  
@view_config(route_name="integrated_user_focus", match_param=
'user_type=B')
 def view_type_b(self, request):
pass







-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/351ad616-1ba0-4f16-b20d-4b8a59590230%40googlegroups.com.

[pylons-discuss] question regarding url dispatch with very similar routes

[Jonathan Vanasco]

I hope someone here can suggest a good way of handling this situation.

We have an application that supports a given user profile view via two or 
more possible URL patterns:

   "/users/{username_regex}"
   "/users/{userid_regex}"

To handle this within one route declaration, a `factory` is used:

config.add_route("user_focus", "/users/{user_identifier}", 
factory=factory__useraccount)

The factory does regex on the `user_identifier` placeholder, and updates 
the request matchdict with an extracted `user_name` or `user_id`. this 
works perfect and allows us to avoid this:

config.add_route("user_focus_a", "/users/{username_identifier}")
config.add_route("user_focus_b", "/users/{userid_identifier}")

we also avoid using view_config to register 2 routes onto each callable


We've added some new functionality, and I'm having a problem figuring out 
the best way to implement it.

users matching a particular new format of the `user_identifier` placeholder 
need to respond to another route

there are essentially 4 potential patterns

config.add_route("usertype_a_focus_username", 
"/users/{username_A_regex}")
config.add_route("usertype_a_focus_userid", 
"/users/{userid_A_regex}")
config.add_route("usertype_b_focus_username", 
"/users/{username_B_regex}")
config.add_route("usertype_b_focus_userid", 
"/users/{userid_B_regex}")

ideally i'd love to simplify this into 2 routes, as they map to 2 views. 
something like:

config.add_route("usertype_a_focus", 
"/users/{usertype_a_identifier}", ...)
config.add_route("usertype_b_focus", 
"/users/{usertype_b_identifier}", ...)

the problem with the approach is that `usertype_a_focus` grabs the route,  
and the `factory` (to my knowledge) can only raise a not-found - not fail 
the lookup and continue on to future patterns - so `usertype_b_focus` never 
runs.

it looks like I could do this by running a regex in the placeholder, but 
then I'm running it again in the factory to alter the matchdict. that will 
also require duplicating an already complied regex into the pattern syntax 
for the route.  the app is large, so I'd like to avoid having to keep two 
different regex patterns in sync with one another.

does anyone know if it is possible to better use factory, or implement some 
other amazing Pyramid utility like predicates or something else in this 
situation?

thank you all in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/d603ef95-b391-43e2-8c2f-82132fc1f952%40googlegroups.com.

Re: [pylons-discuss] most performant way to run WSGI app as single process

[Jonathan Vanasco]

On Monday, September 16, 2019 at 8:37:17 PM UTC-4, hynek wrote:

- HAProxy kicks nginx’s and Apache’s behinds in almost every regard. This 
> is my hill. I like my hill. I will retire on this hill. 
>

I really like HAProxy and Varnish. They were both keystones to some high 
traffic sites I worked on.

I mostly switched form nginx to the OpenResty fork a few years back. The 
"theoretical performance" isn't as good as HAProxy, but the "actual 
performance in real situations" has been negligible.  Being able to program 
at hooks into the various HTTP/nginx request cycle has been much faster to 
develop and deploy various product features for the teams I worked with.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9ffb8639-74b3-4823-a2cf-b8a01405803f%40googlegroups.com.

Re: [pylons-discuss] most performant way to run WSGI app as single process

[Jonathan Vanasco]

On Wednesday, September 18, 2019 at 1:34:57 AM UTC-4, hynek wrote:
>
>
> This means "recycle after 4096 +/- 128 requests” (The jitter is so they 
> don’t recycle all at once; although it’s unlikely to happen at the same 
> time over the whole cluster. The option is more useful when you have more 
> than one worker.).


That I knew... you actually answered what I needed to know above... with 
this line

On Wednesday, September 18, 2019 at 1:31:01 AM UTC-4, hynek wrote:

> Eh, yes it’s true that I run actually two: the gunicorn master and one 
> child process as worker.


I didn't realize that's how 1 process worked in gunicorn; i assumed it just 
ran everything in master.  that's an interesting approach.  i'll definitely 
consider it when we do a devops sprint next!

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/fa622c6b-f67f-4c07-87c6-1c98a0403fe6%40googlegroups.com.

Re: [pylons-discuss] most performant way to run WSGI app as single process

[Jonathan Vanasco]

On Monday, September 16, 2019 at 8:37:17 PM UTC-4, hynek wrote:
 

> - I don’t know whether it’s me, my Python DB driver (sqlanydb ), or the 
> underlying libs: there’s stuff leaking all the time so I wouldn’t use a 
> WSGI container that doesn’t do worker recycling after a configurable amount 
> of requests served. Otherwise you get best case uncontrolled recycling via 
> crash and worst case deadlocks.
>

this is actually the main reason why I adopted uwsgi. a hook loads most of 
the Python we need into the shared memory before the fork, and then workers 
are recycled to our specs.

how do you avoid leaks / leak-like process growth in the threaded gunicorn 
solution you adopted? 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5df47be5-5f1a-40cc-870b-9e5132925434%40googlegroups.com.

Re: [pylons-discuss] most performant way to run WSGI app as single process

[Jonathan Vanasco]

On Thursday, September 12, 2019 at 1:54:53 PM UTC-4, Theron Luhn wrote: 
>
> Usually you put an HTTP server like nginx or HAProxy in front of your WSGI 
> server, which are very good at handling high-volume HTTP requests, so that 
> your WSGI server isn't subjected to whatever the internet might send your 
> way.
>

in addition to that:

* I've found a forking server like uwsgi / gunicorn to be the most 
performant strategy for most situations, as their "max_requests" settings 
to restart workers will eliminate most issues with process growth or memory 
leaks.  they both also offer numerous deployment and management benefits

and in addition to what Bert said...

every deployment strategy has a pros and cons that are from tradeoffs in 
the design of the infrastructure. there is no overall "best option". your 
code could run blazingly fast on one stack with specific resources, and run 
terribly on another.  

while i have a handful of apps that could redeploy using a single-process 
strategy with no impact, moving my main app from a forked multi-process 
model to a series of single-process containers would greatly increase our 
cloud computing bills (we tested it a while back, and it would have 
required way more servers/resources).

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/4459efb5-7ba5-487d-8409-3089ad3c308b%40googlegroups.com.

Re: [pylons-discuss] Velruse… or what else?

[Jonathan Vanasco]

On Friday, July 26, 2019 at 1:14:08 PM UTC-4, Jens Troeger wrote:
>
>
> Regarding Pyramid and OAuth2, there are a few projects, e.g. 
> pyramid_oauth2_provider 
>  (last updated 
> Jun 2017), pyramid_oauth2_client 
>  (last updated Feb 
> 2012), pyramid-oauthlib 
> 
>  
> (last updated Jun 2019). Other projects like apex 
>  seem also unmaintained. Then there is 
> pyramid_fullauth  (last 
> updated Mar 2019) which seems to support an interface to use auth with 
> other providers but I can’t quite tell if they’re built in.
>

I opensourced pyramid_oauthlib_lowlevel 
 a few months ago, 
but it's been used in production for a few years. It is a very lowlevel 
implementation of the oauthlib library that may be more adaptable to your 
needs than the other libraries.  It lets you very quickly put together 
custom oauth APIs and endpoints for Pyramid + SqlAlchemy, but doesn't 
auto-generate any routes/views itself.  If you're only talking with one 
provider or offering one api, one of the other libraries is likely better, 
as they do the automatic route generation.  

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0f9b3199-bb41-4809-8029-45326a6e42cc%40googlegroups.com.

Re: [pylons-discuss] waitress: can not see the 4 default threads

[Jonathan Vanasco]

On Thursday, July 18, 2019 at 6:52:23 PM UTC-4, Michael Merickel wrote:
>
> processes != threads
>

ack! yes! sorry! 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/bd6607d0-1b44-4bbd-a35d-9044df048bf2%40googlegroups.com.

[pylons-discuss] Re: waitress: can not see the 4 default threads

[Jonathan Vanasco]

You could have configured the process to only use 1 thread. You could also 
be checking in the same thread, not a dispatched thread.  There are other 
possibilities too.

It's not really possible to answer a question like this without seeing code 
that reproduces this.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a1d8075f-82b6-40d4-a4df-28a148fecdc7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: How to integrate spatialite in pyramid?

[Jonathan Vanasco]

1. A quick online search shows this tutorial on spatialiate with 
sqlalchemy:  
https://geoalchemy-2.readthedocs.io/en/latest/spatialite_tutorial.html

the author is using a SqlAlchemy event to load the spatialate extension 
into the engine/connection/pool.

2. You are going to have a small problem with this bit:

 Local database: sqlite (for spatialite)
 Remote database: postgresql

The Pyramid cookiecutter and tutorial is designed for a single database 
connection.  Here, you want two database connections .

An easy way to do that is to make a second version of the database 
create-engine, and also duplicate the bit about adding the database to the 
request.  
It's not particularly hard, but it is a fair amount of work and one of the 
more advanced use-cases.  IIRC, there is another approach where you can 
configure SqlAlchemy to shard certain tables onto a first connection, and 
shard the other tables onto a second connection.  That approach is  a bit 
more difficult.

There may be a tutorial on that somewhere.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/14801cca-a6ff-4b38-913e-33f340012eb9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Is there a projected or likely Pyramid 2.0 release date yet?

[Jonathan Vanasco]

On Thursday, June 27, 2019 at 3:33:42 PM UTC-4, Steve Piercy wrote:
>
> No date, but progress is being made.  Check the issue tracker.  We use 
> milestones. 
>

Thanks Steve. I knew about that, but some of those items are very large - 
especially the "possible 2.0 list".

Perhaps I should have been more vague in my question... are there any 
"likely" or "unlikely" timeframes for releases?  As in "most likely not 
until this winter" or "next spring" or "we're hoping for this fall!"

I'm working on pinning the package requirements in anything I maintain to 
"pyramid<2" based on the current deprecations/compatibility items in the 
changelog.  I didn't really think of the changes breaking the pyramid 
ecosystem until someone mentioned a removed API method in a ticket today.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/eda98fc6-430a-40e8-bd97-e787e9262acb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Is there a projected or likely Pyramid 2.0 release date yet?

[Jonathan Vanasco]

I couldn't find anything in the docs or issue tracker.  

I'm wondering how much time I have to create 2.0 compatible versions of 
some libraries.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/8a923ec2-bf1d-4f61-92ae-9f87416c85f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

On Friday, June 21, 2019 at 8:43:32 PM UTC-4, Mike Orr wrote:
>
> But if  I want to contribute to the enterprise's Single Sign-In, do I need 
> to 
> tell the server the user is still logged into my application so it 
> doesn't expire the SSO account? 


That is up to your upstream identity provider.

Your token should be assumed valid until the expiry timestamp. The upstream 
provider may have it's own protocol where you are supposed to check 
validity at periodic intervals.

 

> Do I do this by refreshing the token? 
>
No.
 

> If I do want to refresh the token, do I do it in a NewRequest subscriber? 
>
Generally this is done by a background process.  If you want to do it 
within a Pyramid request, you could do it in a tween or subscriber.

The refresh_expire is 60 minutes, so how close to the end should I do 
> the refresh? If a request comes in 10 minutes before the end, I don't 
> know whether the next request will be in 1 second or 20 minutes. 


This is largely an organizational issue on how long you can honor the 
tokens.  TBH, expiring a refresh in 60 minutes sounds like a bad 
configuration.  Typically an access token would expire in 60 minutes, but 
the refresh token would expire in 30 days.  That would mean you would need 
to automatically refresh the access token behind-the-scenes every 60 
minutes.

 

> What  if the request contains POST data? Would I have to save the data in 
> the session, generate an authorization URL, redirect to the server, 
> come back through the callback view, redirect back to the original 
> URL, and extract the POST data in the session. (Where it's no longer 
> in request.POST.) That sounds like a lot of code overhead. 
>

If you need the user to re-authorize because your refresh token expired... 
you would need to stash the POST data, so that is one way of doing it.  
That is a lot of overhead, but should not happen because the refresh token 
should live longer.

I may also have to put a session lock in the site; i.e., Javascript  

that waits for an idle timeout and puts up a modal dialog, "Do you 
> want to extend your session?" or "Your session is expired. Click here 
> to log in again." How would that interact with tokens and refreshing 
> tokens, since my token processing is in the backend? 


If you have csrf field on the form, that csrf token might have changed. You 
could update it with javascript.
 

> How could the  Javascript extend the session or have the user log in again 
> without 
> throwing away a partially-filled-in form? 
>

Do the reauth in an ajax request or a browser popup.
 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6eb8c654-073d-428d-b404-2fb22d751092%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: How to handle SQLA autoflush failures during Request commit?

[Jonathan Vanasco]

This generally happens because of an issue in your exception handling or 
because you are trying to share a database connection across process 
boundaries. 

The first thing to do is to check if you are connecting to the database 
before gunicorn forked.  If so, this will likely go away if you call 
`dispose` on the SqlAlchemy engine.  `dispose` will essentially reset the 
connection pool, and ensure each worker will have it's own, dedicated pool

There could also be an issue where you are sharing a Session across 
requests or process boundaries.  that can be harder to pinpoint.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/7cc14061-1c9f-4980-b5eb-722c723240f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: How to get an input's ID and value in Deform sequence of mapping widget with jQuery?

[Jonathan Vanasco]

I don't know Deform, but have done some similar things in the past so may 
be able to offer a hint or two.

1. You can grab the container that all the form elements are in in jquery - 

var dom_container = $("#item-deformField1");

2. then, within that node, you can grab each group... ".deform-seq-item"

3. and within each group, you can find the name/age using a jquery selector
'type="text" name="name'

In terms of detecting a change, that's tricky because the dom elements I'd 
want to bind triggers to are already used.  The best way would be to bind a 
keypress event to the fields, but they're automatically created by clicking 
the add-person button.  

I would create a custom function that binds your keypress function to 
detected text fields, then invokes the `deform.appendSequenceItem` 
function.  I think this may be doable by binding a setup function to the 
AddPerson button to overwrite it, and also invoking the proper 
`deform.addCallback` yourself   
https://docs.pylonsproject.org/projects/deform/en/latest/widget.html#widget-javascript

There may be a better way.   I hope there is!  This just popped out to me.

What looks to be relevant bits of the dom structure it makes is below:



 
  People 
 
 
 
 
 
 
 

 
 Add Person
 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/873f0986-043d-40e2-8966-e82c4f24785e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

Many of the things you mentioned are specific to Keycloak.  I can't answer 
those.

I can take a deeper look at the oauth details tomorrow.  the following are 
things that just popped at me.

On Thursday, May 30, 2019 at 8:57:28 PM UTC-4, Mike Orr wrote:
>
> - What should I do if there's a state mismatch? I don't want to give 
>
the user an Internal Server Error, or tell them they're bad when 
> there's nothing they can do about it. 
>
 
If there is a state mismatch, something very bad happened and you should 
not let them login.

- What should I do with the token? 
> - Should I save the token in the session? 
>

Are they using oAuth to login to your app every time?  If so, you should 
save it in the database.  If they will authorize via oAuth every time they 
use the app, you could store it in the session or trash it.
 

> - What's the difference between the token keys 'expires_in' and 
> 'refresh_expires_in'? 
>

expires_in the expiry of the access_token.  the auth server will not 
recognize the access_token after that point.

refresh_expires_in the expiry of the refresh_token.  the auth server will 
not issue a new access_token in exchange for the refresh_token after that 
point.

- Should I refresh the token? Where would I do that? In every view 
> callable? My existing LDAP implementation doesn't have this concept; 
> it leaves the user logged in until the Pyramid session expires and is 
> deleted in Redis. 
>

You only refresh the token when it is about to expire.

When a user approves your app, the server gives you two secret tokens.
You use the access token to make all your requests to the server.  That is 
likely saved in the database and the user's session.  Think of it as a 
"session_id" for the upstream api.
You keep the refresh token tucked away in the database. You only use that 
token to ask for new access tokens.  Think of it as an autologin cookie to 
the upstream api.

- I'm instantiating the OAuth2Session in each view. Should I save it 
> somewhere? I can't put it in the Pyramid session because it's not 
> JSONable. Is it expensive to reinstantiate it like this? 
>
>  
You should only require/create it on the login and callback.  At the 
callback, if everything is successful, you can log the user in as a user of 
your application.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9535c01f-d656-44ad-a81b-5538b4f57c32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] RFC: Pyramid showcase of RealWorld.io

[Jonathan Vanasco]

On Sunday, May 26, 2019 at 6:17:53 AM UTC-4, hynek wrote:
>
>
> t I’ve started putting separate features into separate sub-packages and it 
> made my maintenance experience much more pleasant thanks to the locality of 
> common contexts. As for tests, I just mirror the structure in my global 
> tests directory. That’s probably just a matter of taste.


The biggest improvement we had to overall efficiency and happiness was 
splitting the model into it's own package.  We have multiple Pyramid, 
Python, Celery, Twisted, etc apps all import a unified model.  The model 
has it's own tests, each package/application has it's own tests against the 
model as well.  Doing this centralized and simplified the management of the 
database across all the different components, and more importantly multiple 
levels of testing.  The end result was a much more frictionless 
build/deploy.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5f8b2416-76ef-482d-94ce-8d12758cad03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

On Tuesday, May 21, 2019 at 8:13:11 PM UTC-4, Mike Orr wrote:
>
>
> I want it to do username/password checking and give me the user 
> metadata including the list of roles. Then I can do authorization from 
> the roles. By "no authorization" I mean the role/view ACLs would 
> remain in Pyramid; I don't want to put them into Keycloak and ask it 
> every page view whether this user has permission to this view. 



 

> Yes, that's what's so complicated, trying to figure out which classes 
> I need. I'm trying to implement the "Web Application Flow" in the 
> 'requests-oauthlib' docs. So I would redirect to Keycloak, and it 
> would authenticate the user and redirect back. I'm not sure what 
> exactly to do after that. Keycloak would manage the user's account, 
> password, whether the user can access this application, and a list of 
> roles related to the application. 
>

Okay. Within the realm of oAuth and Identity Management, Keycloak is 
handling Authentication of the user and your Pyramid App will be delegating 
Authorization to it.  Everything else you're talking about in regards to 
"Authorization" and interpreting the ACLs is within Pyramid and a slightly 
different concept of Authorization.  In terms of oAuth... Pyramid is a 
client and using Keycloak for Authorization, even though it seems like 
you're using it for Authentication.

Let me talk about a few things you brought up of order, and then try to 
answer some questions while I leave you with more...

Authorization Headers

This is fully compatible with Pyramid, don't worry. The oAuth specs 
prefer/suggest/request that certain oAuth information is exchanged within 
request headers and not within query strings or post data.  This data 
exchange generally happens between different servers and is implemented by 
the oauth client (likely requests-oauthlib) or the upstream authority 
server.  It's an implementation detail that is really unrelated to your 
usage.  IIRC, the two places this happens are in the post-authorization 
redirect from the Authority to your Callback URL and then again in a 
backend request from your server to the Authority.

oAuth Flows and Grants

Forget about classes in libraries and just focus on the Spec/Use-Case. I'm 
pretty sure you want the standard Authorization Code grant, but just to be 
sure here If you search "oauth flows", you'll see a bunch of blogs and 
resources describing the different grants and flows.  Some of the things 
you wrote about sound like the legacy Resource Owner Password Credentials 
flow; if that's what you're trying to do... i suggest you take another look 
at the Authorization Code flow

* https://nordicapis.com/8-types-of-oauth-flows-and-powers/
* https://oauth.net/2/
* https://auth0.com/docs/api-auth/which-oauth-flow-to-use
* 
https://medium.com/@darutk/diagrams-and-movies-of-all-the-oauth-2-0-flows-194f3c3ade85
* https://requests-oauthlib.readthedocs.io/en/latest/oauth2_workflow.html

Let me explain a bit of the Authorization Code Grant using my library, 
because that sounds like what you should be doing (that flow, not my 
library)...

1. The user visits your Application, and is redirected to the Authority 
with a payload which contains the application id, a random id, a set of 
credentials to authorize, what they want, and callback URL that you already 
configured as valid with the authority.  At the Authority, they will either 
login or be logged in, and Authorize the grant.

 This Pyramid view is a user being redirected from your application: 
https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/master/pyramid_oauthlib_lowlevel/tests/oauth2_app/views.py#L654-L671

2. At the Authority, they generate a temporary oAuth Grant Token to track 
the authorization and redirect the user to a predetermined callback page on 
your application.  There is some oAuth information in the headers of the 
redirect. 
 This is the user at the Authority view- 
https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/master/pyramid_oauthlib_lowlevel/tests/oauth2_app/views.py#L178-L241
 The GET params on the request to the authority will have a clientId 
(your app), a redirect url (which must match the one in their database) and 
a 'state' nonce/session
 The authority allows the Authenticated user to Authorize the 
credentials request request, create a session on their side, then redirect 
to your page.

3. Your application processes the oAuth information contained in the 
headers on the callback page, and then (4) redirects to a success page or 
the page originally requested.  A library will handle processing all this 
stuff for you, you should just be pulling the information from validated 
request via the library's API.

This is where that happens 
https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/master/pyramid_oauthlib_lowlevel/tests/oauth2_app/views.py#L673-L724

While this is going on, there are some behind the scenes communications 
between your server and the 

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

Thanks for the kind words.  This was actually pretty fast to do.  I think 
it took 2-3 days to build oAuth into our apps and almost everything was 
repackaged into this within that same week. This has been sitting in a 
private repo for a year or so, because it required some updates to oauthlib 
and requests-oauthlib that I wrote but hadn't been merged/released yet.  
The tough bit was getting the testing harness set up right. 


I had written a lot of stuff explaining some things for you, and then 
realized I may be all wrong in terms of what you're trying to accomplish.  
It reads like you're trying to do a lot of different authorization things, 
but then you specifically said no authorization but authentication.  So I'm 
getting tripped up a bit, especially by this line:  "I don't need 
authorization: just authenticating the user and retrieving their roles and 
metadata and maybe the refresh feature"

If the roles and metadata for the user are in Keycloak, then Keycloak is 
handling the Authentication (e.g. the user authenticates to keycloak by 
logging in) and you actually want Authorization to happen (which is what 
you described above).

So let's pull back for minute and get a better idea of what you're trying 
to do in terms of use-cases.

What is the intended interplay of your Keycloak server and your Pyramid 
application?  

oAuth can be very annoying because one spec/project/initiative/framework 
covers a handful of ways to implement it.  The different implementation 
methods are called "oAuth Flows" or "oAuth Grants".  The first think you 
need to do, is to decide exactly which grant you want do implement.  
Depending on how you look at it, there are  between 4 and 8 flows covered 
by a handful of RFCs, all under the umbrella of oAuth. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e549d218-81ac-4762-b8cc-90e53313bf86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

mike-

if you haven't figured it out yet, hopefully my examples can help:  
https://github.com/jvanasco/pyramid_oauthlib_lowlevel 

i've been meaning to release this for a while now, thanks for giving me a 
reason to.

in the tests, you can see a full flow of interacting with clients and 
servers:

https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/master/pyramid_oauthlib_lowlevel/tests/oauth2.py

the client and server views are implemented in this file:

https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/master/pyramid_oauthlib_lowlevel/tests/oauth2_app/views.py

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/75af89f8-fb92-40e7-a014-aca324943e17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Authenticating with Oauth2/OpenID Connect

[Jonathan Vanasco]

I have a project that I've been meaning to Open Source.  I'll try to make 
time tomorrow to repackage it... it's called pyramid_oauthlib_lowlevel and 
is a VERY LOWLEVEL integration tool for oauthlib against Pyramid, it was 
somewhat inspired by flask-oauthlib and there are example oauth1 and oauth2 
system in the test suite.  

Something that may be confusing you, is that you essentially do need an 
oAuth server.  There are really three main ways to integrate oAuth :

* Server: Identity Authority
* Server: Client Application
* Developer Client

The "Client Application" and "Developer Client" are very similar in that 
they both authenticate against the Identity Authority, but they typically 
use very different methods of handling a credential exchange.  You can 
implement both with the requests-oauthlib library.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a215733c-5bc0-407a-974d-2e5063deda88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid 1.10.4 released

[Jonathan Vanasco]

On Monday, May 13, 2019 at 4:17:05 AM UTC-4, Steve Piercy wrote:
>
> Done! 
>
> https://pylonsproject.org/community-support.html 
> https://trypyramid.com/documentation.html 
>
> Thank you for bringing up the questions. 
>
>
I think it would make sense to put a formal policy in place requesting 
security issues be first reported via the @security email address, they 
will be in contact within 72hours or so to discuss if any next steps are 
appropriate, and to please refrain from public disclosure or creating 
report with CVE within this timeframe. 

SqlAlchemy got hit with a few CVE's recently over some documented 
behaviors; because the reporters filed a CVE first, it set off a cascade of 
alerts across multiple projects that use it and packaging systems. The 
process was a lot more stressful for the maintainer than it should have 
been.



-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/084317fd-5fee-45f2-94af-fde41518628e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Questions about DB engine setup

[Jonathan Vanasco]

> As I understand each worker needs it's own engine instance, don't they? I 
think the gunicorn behaviour is good, but I'm puzzled by the 
pserve/Waitress behaviour. Is this by design?

Just to expand on Michael's comment on there usually only being one engine 
but multiple connections...

Worker's don't need their own engine instances, but they need their own 
connection pools.  gunicorn (and uwsgi) implement a forked process model, 
and the database connections are not safe to move across the process 
boundaries due to some things inherent to Python and/or filesystems .  if 
you connect to the database during setup in this model, you need to call 
`engine.dispose()` when setup is complete or after the fork, or else weird 
and bad things will eventually happen.  see 
https://docs.sqlalchemy.org/en/13/core/connections.html#engine-disposal .  
 waitress is multithreaded and the pool is supposed to be safe across 
process boundaries. 

You should be able to just do an Engine on the request, but if you do 
ensure you do the `dispose()` or set up an event to ensure process safety.  
it's all the pooling docs 
https://docs.sqlalchemy.org/en/13/core/pooling.htmlyou'll also have to 
implement your own transaction management, which is honestly not fun.  

if you are sure you're only doing SELECTS... you could have a dedicated 
engine/connection string for a read-only database user and have that bypass 
the transaction system entirely.  i use that pattern a lot, but you need to 
make sure you enforce it as read-only at the database credentials level or 
you have the potential for a lot of issues bubbling up.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0e53f127-d115-4724-95aa-5cfd14f23223%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Questions about DB engine setup

[Jonathan Vanasco]

On Saturday, April 27, 2019 at 11:43:33 PM UTC-4, Mike Orr wrote:
 

> 'session.execute()' executes SQL like 'engine.execute()' does, so you 
> can get the advantages of a request transaction without using ORM 
> queries. 
>

in addition to Mike's comments - make sure to read the zope.sqlalchemy docs 
on `mark_changed` or starting the session on a 'changed' state

https://pypi.org/project/zope.sqlalchemy/


if you don't do that, sqlalchemy won't know you did something and will 
rollback instead of commit.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6015d75c-f277-42cb-880c-9d30632fdcb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] pyramid_csrf_multi_scheme v0.0.7 released; feedback welcome

[Jonathan Vanasco]

I updated pyramid_csrf_multi_scheme to work with the 1.10 line of pyramid 
on github (https://github.com/jvanasco/pyramid_csrf_multi_scheme ) and pypi 
(pyramid_csrf_multi_scheme)

the package is designed for people who need to support csrf in a mixed 
scheme environment, and offers `DualCookieCSRFStoragePolicy` which 
reimplements Pyramid's `CookieCSRFStoragePolicy` with two slight 
differences:

* there are separate HTTP and HTTPS csrf cookies.  expiring the HTTPS will 
expire the HTTP, otherwise the HTTP is only accessible on HTTP connections 
and HTTPS on HTTPS connections
* `cookie_profile` is replaced with `cookie_profile_secure` and 
`cookie_profile_http` 

If you need csrf on a mixed scheme environment, you may find this package 
helpful.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6542c2f4-da47-4a85-be2e-d37fa3de934b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Pyramid 1.10.3 released

[Jonathan Vanasco]

On Friday, April 12, 2019 at 1:30:21 AM UTC-4, Michael Merickel wrote:
>
> Hey folks, I just released 1.10.3. The changes are small but it does add 
> the ignore_files setting to [pserve] which some people may appreciate.
>

WOO HOO! thank you! 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/4aac128c-54d9-4926-8f5a-c33c39cf02f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] `reload` detecting changes to mako templates (again)

[Jonathan Vanasco]

On Friday, March 29, 2019 at 8:24:27 PM UTC-4, Bert JW Regeer wrote:
>
> Mako templates are compiled and imported, when that file is updated the 
> reloader likely sees it, unless it is detecting the file changing on disk.
>
> For the former: I am not sure how to necessarily stop that from 
> happening... but
>
> For the latter I would try the new ignore files feature that Michael added 
> to hupper: https://github.com/Pylons/hupper/pull/46
>

Oh, this looks GREAT. I've been hoping for a feature like this for a long 
time.  thank you Bert, and Michael.

Mako, along with Jinja and some other libraries, have their own change 
detection and reloading systems.  Every so often, a new random library that 
Pyramid has a dependency on will have an update that recognizes previously 
un-included directories.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/c5ab298d-ce2a-44ed-b21c-ca5cb18d10ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] `reload` detecting changes to mako templates (again)

[Jonathan Vanasco]

This happens every so often, but on a large project is making editing the 
HTML incredibly tedious.

Does anyone have a good strategy for tracking down which component is 
breaking this and effectively whitelisting the /data directory

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/4707172a-42d0-4ec0-9822-a2b1a0783e42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] is there a standardized way of detecting the environment (e.g. bootstrap)

[Jonathan Vanasco]

On Thursday, March 21, 2019 at 12:57:36 PM UTC-4, Mike Orr wrote:

>
> If the reason you want to know whether you're running under bootstrap 
> is to determine whether logging has been configured, 
>

We log every startup/invocation of the app/scripts to a centralized 
database.

There's a table that lists every startup with a timestamp and the context - 
uwsgi/waitress workers, prequest, and bootstrap, along with a node 
identifier for the machine and some other data.  

It is one of the systems used for healthchecks, to ensure everything is 
running properly and help auditing during post-mortem incident analysis.  
The raw logs often get cycled into oblivion or destroyed, so this helps us 
pinpoint where to dig in to data first by detecting when cronjobs stopped 
running or when applications spiraled into endless restarts.  Other tools 
ensure services are running or jobs have completed, this is just one useful 
component of monitoring.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9e924119-54ba-4b55-915c-58067c48f66b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] is there a standardized way of detecting the environment (e.g. bootstrap)

[Jonathan Vanasco]

On Wednesday, March 20, 2019 at 2:22:12 PM UTC-4, Michael Merickel wrote:
>
> The hackiest option possible is to just set a global variable somewhere 
> when you use bootstrap. Otherwise you could use a custom setting.
>

Oh, I can think of hackier ways!

One thing I considered was overriding bootstrap and pshell with a custom 
package, which just sets a global variable and then provides the actual 
underlying tools.  I still have to go through all the ways of running a 
pyramid application.  I was originally leaning towards custom config files, 
but worry about correctly supporting the wide range of config options.

The use-case for the plugin i'm trying to build is centrally logging 
how/when multiple pyramid processes run to monitor health.  One project, 
for example, has three pyramid apps running and about a dozen cronjobs 
running bootstrap or prequest.  It's currently using ENV variables to 
control the logging, but it's very messy.  I'd like to standardize this 
into something that can cleanly run for all our projects and be released on 
PyPi.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/bb2ff0c6-58ca-43b2-9777-5926e3e36189%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] is there a standardized way of detecting the environment (e.g. bootstrap)

[Jonathan Vanasco]

Thanks for the quick reply, Michael.  I'll have to ponder a solution to 
this problem for a bit.


On Wednesday, March 13, 2019 at 3:53:54 PM UTC-4, Michael Merickel wrote:
>
> No bootstrap does not inject any custom settings or anything into the app.
>
> On Wed, Mar 13, 2019 at 2:33 PM Jonathan Vanasco  > wrote:
>
>> there seems to be a few ways a pyramid app can be run
>>
>> is there a way to detect if the current application was started via 
>> `pyramid.paster.bootstrap`? 
>>
>> i'm trying to sketch out a (public) plugin that can log application 
>> startups and invocations.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com .
>> To post to this group, send email to pylons-...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/2329b3cc-55d8-4aca-a19d-2f3f4ff0ed5f%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/pylons-discuss/2329b3cc-55d8-4aca-a19d-2f3f4ff0ed5f%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/17a8d092-f1ac-4bee-84f5-81241187a5a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] is there a standardized way of detecting the environment (e.g. bootstrap)

[Jonathan Vanasco]

there seems to be a few ways a pyramid app can be run

is there a way to detect if the current application was started via 
`pyramid.paster.bootstrap`? 

i'm trying to sketch out a (public) plugin that can log application 
startups and invocations.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/2329b3cc-55d8-4aca-a19d-2f3f4ff0ed5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pylons Project Assimilation team meeting minutes from 2019-02-25

[Jonathan Vanasco]

Thanks for the heads up. This all makes perfect sense. It was just weird 
seeing "minutes" posted with no previous announcements by this.  i thought 
I missed something and couldn't figure out how!

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/54669d73-0628-446b-9bd5-8d1ba8a01de1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Pylons Project Assimilation team meeting minutes from 2019-02-25

[Jonathan Vanasco]

can you share anything on this migration/etc?

there have only been two posts about this as minutes, but no other 
announcements that I could find.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/84db2a78-7f8e-44fb-bfb7-1dab40009d9f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Testing Pyramid Traversal App with ZODB

[Jonathan Vanasco]

We use SqlAlchemy but perhaps this will still help:

We test anything dealing with transactions via Functional Tests or 
sometimes Integrated Tests, not Unit Tests.

For Functional Tests, look for examples in the docs that are based on `from 
webtest import TestApp`.  I think I have an open source project or two that 
does this, and can check.

Basically, we  construct the test harness to create the database on setUp, 
making it available until tearDown.  Then you invoke the logic by a route 
which calls the function, so a test looks something like this...


def test_edit(self):
  res_new = self.testapp.get('/item_new', status=200)
  res_item_id = 1  # or regex the item id off the response
  dbItem = 
self.ctx.dbSession.query(model.Foo).order_by(model.Foo.id.desc()).first()
  assert item_id == dbItem.id

if you're doing an edit flow, then you need to use a new dbSession (or 
clear the existing one) so the stale data doesn't persist.  This works fine 
with testing via sqlite, mysql and postgres

the general concept though is...

* test harness creates a new database
* the app uses the new database
* queries in the test harness use the new database

we also have some Integrated tests that spin up an instance of the Pyramid 
app, then make queries against it using a headless browser.

to the best of my knowledge, those are the only ways to really test 
anything involving the transaction package 


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/b946ab4e-d480-4516-9eae-00a9d245ae4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: GSOC 2019

[Jonathan Vanasco]

there are some GSOC brainstorming/operations/etc docs from other years that 
can help you get started:  https://github.com/pylons/pyramid/wiki


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/75313fa0-2fa4-4a56-a9fc-dd7d23cca0e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Convert Response to HTTP, and a JSON error response

[Jonathan Vanasco]

To accomplish something similar, i use a mix of custom exceptions and 
pyramid's exception handling.

I have a class in my exceptions.py like this...

class ViewExceptionJsonForm(Exception):
def __init__(self, ... ):
... setup with args, which might contain the Form, a Wrapped 
Exception, etc

In my view I would raise an error...

@view_config(route="my_view")
def my_view(self, req):
...
if form.errors:
raise ViewExceptionJsonForm(form=form)

And then I map it to a custom exception handler, which does everything for 
me...

@view_config(context=ViewExceptionJsonForm, renderer='json')
def exception_view__JsonForm(exc, request):
request.response.status_code = xyz
rval = new_rval()
rval = {...}
# populate rval based on `exc`
return rval


Whenever I have an issue processing an API form, I raise this exception... 
and then the pyramid internals handle everything else. It's pretty great!

But as Michael noted, you probably have an issue with the frontend not 
having an appropriate accept header (or the request is being made for a 
non-json route/context if you're supporting stuff like that).

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/58eb16cd-42ec-4299-8d9c-f185c7671fd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] processing after responding to client

[Jonathan Vanasco]

On Thursday, January 10, 2019 at 8:27:35 AM UTC-5, Zsolt Ero wrote:
>
> So I guess the simplest possible solution is Celery + Redis (since I'm 
> already using Redis for sessions), I guess? 
>
>
Yes. The performance you want requires a secondary process to handle the 
event logging. Celery is probably the simplest was, though there may be 
faster implementations (like Thierry's use of 0mq).

Two other options that may work for you (I use both of these, and Celery):

* if you can use statsd for logging, that setup just sends the event data 
via UDP (not TCP) and an external statsd server will handle the logging of 
the event args.
* you can use a secondary database connection / server that is running in 
autocommit mode. that can avoid the normal database, and might be against a 
server that writes faster.  for example, you could write to a mysql 
database that only has logging tables and is tuned for high-performance 
writes.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/17344e53-da39-4af5-a88c-973edc070558%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Waitress 1.2.0 beta 1

[Jonathan Vanasco]

On Monday, January 7, 2019 at 5:00:43 PM UTC-5, Jonathan Vanasco wrote:
>
>
> It would be nice if that functionality could be implemented as middleware 
> when not running the waitress server.
>

This release broke our dev systems from the wsgi manipulation. The issue 
was this line to the nginx config from the instructions:

proxy_set_header X-Forwarded-Host $host:$server_port;

While the the `:$server_port` bit appears in many tutorials and docs on 
some projects, it isn't part of a standard or a standard/requirement.  Even 
nginx's official docs has conflicting usage patterns with it:

* https://www.nginx.com/resources/wiki/start/topics/examples/likeapache/
* https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/

I can't tell if that bit was required by your code. It doesn't look like it 
is required (via `task.py` lines 531+) but perhaps a warning about that 
would help. 

Sidenote: it looks like the strip on line 680 may be better placed just 
above the immediate conditional


-if forwarded_host:
-
-forwarded_host = forwarded_host.strip()

# might be better as...

+forwarded_host = forwarded_host.strip()
+if forwarded_host:



My initial feeling on this release is that I dislike this implementation.  
This approach will create a lot of added complexity for those who use 
waitress for Pyramid in development BUT run other servers in Production 
and/or Staging - we now have to deal with a deploying something where 
Waitress inherently takes control of this portion of wsgi manipulation in 
one environment, but is not run at all in other environments.  

IMHO, it would be really nice if...

1. this happened within Pyramid (not likely to happen, I know)
or
2. this functionality were exposed as a callable, so other deployments can 
invoke it. This will still create some pain in maintaining dual deployment 
logics, but invoking waitress's logic would ensure parity.

Finally...

for the docs... if using an ini file, the syntax is;

[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 5020
trusted_proxy = 127.0.0.1
trusted_proxy_headers = x-forwarded-for x-forwarded-host x-forwarded-proto x
-forwarded-port



using a quoted format from the current docs will generate an error:

trusted_proxy_headers = "x-forwarded-for x-forwarded-host x-forwarded-proto 
x-forwarded-port"





-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/01761583-c857-412f-80c3-117c6af4f82c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Waitress 1.2.0 beta 1

[Jonathan Vanasco]

On Monday, December 31, 2018 at 4:12:10 PM UTC-5, Bert JW Regeer wrote:
>
>
> Please note that in the future Waitress is going to be more secure by 
> default, and will strip known proxy headers before forwarding them on in 
> the WSGI environ to help protect WSGI applications from accidentally using 
> attacker provider proxy headers and their values.
>

It would be nice if that functionality could be implemented as middleware 
when not running the waitress server.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a807cdd9-072c-4a39-b826-df099dfcb52a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Security Headers + Extras Project

[Jonathan Vanasco]

I may have missed something, but it looks like the 'secure' bit of the 
cookie is in the settings, but not the value.

>From a pyramid standpoint, this would be a step backwards - as the default 
policy is a signed cookie.

My 2¢: your package should implement signed cookies by default... and 
support plaintext and encrypted cookies as options.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/c1de437b-35e6-4ab6-8858-6c85a027c6af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] in search of: pattern for integrated testing that pyramid handles certain exceptions/errors correctly

[Jonathan Vanasco]

we have a suite of tests to ensure that a Pyramid app 'breaks' correctly.  
for example: if connectivity with Redis or a SqlAlchemy server is lost, 
there is some specific error processing and results handled via exception 
views;  some other potential issues are handled via tweens.

the current approach to handling this is restarting the Pyramid app with 
certain connections turned off.  we've been considering using a set of 
"testing-only" views, that trigger the exceptions with specific routes.

does anyone have insight/experience on this stuff they can share?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/caf7e2b9-1dd9-4105-baaf-630ae37cfe27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Re: Pyramid Registry Creates Too many connections To the Same Server with pysimplesoap

[Jonathan Vanasco]

On Thursday, November 15, 2018 at 5:43:56 PM UTC-5, Brian Herman wrote:
>
> Oh cool there is sorry for the spam 
> https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/deployment/forked_threaded_servers.html
> 
>

it depends on the container you run pyramid in - they each have a different 
hook.  there's a way of catching it in waitress, but i forget what that is.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/32579a12-5051-4478-800a-33edafe59cce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Pyramid Registry Creates Too many connections To the Same Server with pysimplesoap

[Jonathan Vanasco]

A few things that jump out at me:

1. it looks like your `PN` class is creating 10 different `Client` objects 
which each start a persistent(?) connection to the SOAP server when 
created. that's a huge factor in overloading the connections.

2. I'm not familiar with your SOAP library. are these connection objects 
threadsafe/forksafe?  If not, you should defer the `connect` to happen 
after the application forks/threads. that will result in even more 
connections.

i would look at creating and disposing of the connections as needed, 
instead of starting up with them.  if you need all of them them on every 
request, i'd rethink the application design and how all the nodes interact 
with one other.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/2ea804d0-1b40-49f2-944a-2f890d92f307%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Store class at process start, and use it to initiate inside function?

[Jonathan Vanasco]

On Tuesday, October 9, 2018 at 12:08:35 PM UTC-4, Bert JW Regeer wrote:
>
> I would disagree, heavily. You want to create your globals once, then 
> fork. This way the memory used by said global can be shared between all of 
> the processes. 


+1

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/45cb52dd-a7f3-41d1-8837-cc2bfc225d87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Ethical Ads on Pylons Project documentation

[Jonathan Vanasco]

Sure. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/da01d2c9-f70e-4536-ac54-22868682a127%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Relationship between requests and Zope transactions

[Jonathan Vanasco]

+1 on being explicit by passing around the request (or something similar). 
there are way too many edge cases when you try to compute the current 
request/transaction, and it's incredibly hard (a nightmare) to write tests 
that work correctly when you rely on the derived/computed methods.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e90800f3-9b1b-48fa-91f5-2c4b9668104b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] creating a simple pyramid application for a test suite?

[Jonathan Vanasco]

another testing question!  

I'm in the process of splitting out our oauthlib integration for pyramid 
(it's much more lowlevel than the existing app) after some design changes 
to make the integrated testing work better (which was my last question). 
Now I'm trying to figure out the best way to test the pyramid bit in the 
new package design. 

The tests are currently written within my app, and I want to migrate them 
to the plugin.

That means I need to create a micro application to handle 3 endpoints, and 
define it within the 'tests' of the plugin.  i.e. instead of having the 
tests invoke a package which is an application, the tests are creating 
their own application to invoke a package that is a support library used by 
the application.

Has anyone seen a package that does this?  I'd prefer to work off a 
reference if at all possible.

thanks,
jv



-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/7386f0d4-485a-40a9-b8e5-f7800d7cbd01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] triggering failures during integrated tests?

[Jonathan Vanasco]

One of my applications provides both an oAuth1 server for authentication 
and client for requesting authorization.

Unit tests are fine, but I'd like to trigger failure on certain phases 
during the integrated tests, and could use some advice.

The types of things I'm hoping to test, are basically shutting off certain 
subsystems or submitting malformed requests. These all have unit tests that 
pass, but they should have integrated tests to ensure the requests are 
failing in the expected ways. 

Because the authorization route makes a behind-the-scenes request to the 
same server for the request_token route, I need to run waitress with at 
least 2 threads.  Since I'm using 2 threads, I can't rely on hitting an 
endpoint before the request saying "(en|dis)able fail mode!" - only one 
thread would get it.

I'm looking at a handful of potential options that range from 
middleware/tweens to subscribers and hiding stuff in __debug__ blocks to 
get optimized out of production.  Before I select a crazed, 
overcomplicated, approach... does anyone have a suggestion for handling 
this sort of testing need?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/87afff84-028e-46dd-a97a-5f6b9d405591%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Re: New Package for Integrating Mongodb

[Jonathan Vanasco]

On Wednesday, August 1, 2018 at 3:41:39 AM UTC-4, Jonathan Mackenzie wrote:
>
> I've updated the package now, 
>
> https://github.com/jonnoFTW/pyramid_mongodb2
>
> Everything is integrated into a single package. And now you can alias 
> database names to make testing easier.
>
>>

wow, the toolbar features are amazingly useful! 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/bb6b2618-6b48-4532-9fe1-93b3ad6919f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: New Package for Integrating Mongodb

[Jonathan Vanasco]

On Saturday, July 28, 2018 at 2:05:05 AM UTC-4, Jonathan Mackenzie wrote:
>
> Lack of thought I guess, I should probably merge them and use a config 
> variable to turn debugging on..
>

IMHO I really like the idea of Pyramid packages that ship with a built-in 
debug_toolbar support.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/70df9995-04ce-47de-aabb-b60fba9a4f9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: What types of resources are appropriate to store in a NoSQL database?

[Jonathan Vanasco]

On Thursday, July 26, 2018 at 2:30:31 PM UTC-4, Benjamin Polen wrote:

> what web resources should I store there.


None. 

On Thursday, July 26, 2018 at 2:30:31 PM UTC-4, Benjamin Polen wrote:

> For example, should html files, jinja2 templates, javascript files images, 
> css be stored in my resource tree?
>

No.

The only time to store those types things in nosql is when you're storing 
"user overrides" that are implemented via a customization or whitelabel 
system and the data HAS to be queried. In those scenarios you can't avoid 
the overhead that nosql lookups and IO cost.

In every other situation, you want to leverage things like:
* built-in caching of template libraries
* filesystem IO optimizations of web servers and operating systems

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/212939ff-b1f3-45e6-8804-a40b8717ce95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: New Package for Integrating Mongodb

[Jonathan Vanasco]

any reason why you decided to split the toolbar into a separate package, 
instead of including it?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/fa5aaa0e-a834-4b71-9e79-2bb16ae185e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] feedback needed for new/rebranded package - g_analytics_writer

[Jonathan Vanasco]

i've replaced an earlier package that writes for the "ga.js" library with 
`g_analytics_writer` - a unified way to generate *simple* tracking 
analytics for "Google Analytics" in Pyramid apps.

It offers a unified API for 'telling the package what to track', then will 
generate optimized analytics tracking code for any of 3 available versions 
(ga.js, analytics.js, gtag.js)

I would love some feedback on the alpha release before I push a regular 
version to PyPi  https://github.com/jvanasco/g_analytics_writer

for pyramid, you just configure environment.ini with:

g_analytics_writer.account_id = UA-123123123-1

include the package:

 config.include('g_analytics_writer')

which will add a `g_analytics_writer` object to the pyramid request.

and update template with (mako):

${request.g_analytics_writer.render()|n}
 
It supports custom variables, ecommerce transactions, simple events, 
sending to multiple trackers and some other things.  it's fully tested and 
works on Python 2 & 3.

This has been a bit of a pain to write, as I had to do a lot of testing to 
figure out how to minimize the amount of external API calls on each 
platform.

This was written to handle our concerns/needs on an upgrade/migration from 
`ga.js`, but I'd be happy to extend the capabilities if I have time (or 
take PRs)

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/75d758ad-990a-4a43-9cae-b6b765c70e4c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] some waitress design questions

[Jonathan Vanasco]

On Tuesday, July 17, 2018 at 7:46:59 PM UTC-4, Bert JW Regeer wrote:
>
> The threads that run the WSGI app are pre-spawned, they wait on a new 
> request to be added to a queue, peel one off, pass it down the WSGI app, 
> and then back.
> ...
> What are you trying to do with a "max requests"?
>

Thanks for all the above. It answers all my questions about how waitress 
works.

Several web servers implement a concept of "max number of cumulative 
requests each subprocess can serve" before that subprocess (thread or fork) 
is terminated and replaced with a new process.

The situation I'm trying to debug has to do with some unpredicted behavior 
and a particular 'constant' variable that is assigned during the 
application initialization, and (should only be) read afterwards.  It works 
fine initially, then seems to 'unset' to a default value after an hour or 
so.  I want to ensure the error isn't being triggered when 'max requests' 
is hit and a new subprocess being spawned.

I don't think it is, but I'd like to be sure.  It's been a pain trying to 
recreate this scenario with uwsgi/nginx and apache/mod_wsgi.

 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0d72eb02-4ef8-4f27-8e9a-616db185156b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] some waitress design questions

[Jonathan Vanasco]

does anyone know if the workers in waitress are spawned as-needed for each 
request, or if they are pre-spawned and answer requests when available?

I think it is the latter.  if so, is there a reasonable chance of having a 
max-requests feature implemented (or would this be possible to kludge 
together myself for local testing)?

i'm trying to recreate some issues that are appearing on other wsgi 
containers to debug within waitress, and am having a hard time.  I think 
the problem is happening due to how the worker processes are launched.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/905cf706-6ac6-4a7b-8afc-1381c2f7623e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Re: noob question about redirections

[Jonathan Vanasco]

On Tuesday, July 17, 2018 at 12:45:26 AM UTC-4, jg...@live.com.mx wrote:

> so, to make this clear to me: the POST made from javascript is not 
> equivalent to a POST made in html, the hmtl POST automatically refreshs 
> page or something like that?
>

This may sound confusing... but their "equivalence" is based on perspective:

As far as your Pyramid app is concerned, they are equivalent
As far as the HTTP Specification is concerned, they are equivalent
As far as almost everything is concerned... they are equivalent

BUT

As far as a user is concerned *in this scenario* they are not 
equivalent.
Clicking "Submit" to post the form has the browser window post the form 
and handle the response;
Clicking your xhr button instructs the browser to do everything within 
javascript - so javascript is then responsible for posting the form and 
handling the response.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/eb63d5bf-c41d-4178-8c5a-14ed956eac53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: noob question about redirections

[Jonathan Vanasco]

the issue is in your javascript; the browser will never render the template 
on this setup.

when you view /test via GET (or missing one or more required POST items), 
templates/jg.pt is rendered.
when you view /test via POST with both variables present, the response is a 
redirect (HTTPFound)

your javascript submits "form data" but it's not submitting the form for 
the browser.  the response from pyramid is lost in javascript.

your flow is this:

* browser-tab GET /test 
** browser-tab's javascript POST /test 
** pyramid redirect to /home
** browser-tab's javascript ???

in this flow, the browser tab doesn't POST, the javascript engine in the 
tab does.

because the POST happens within javascript, you need to use javascript to 
decide what to do.

typically before hitting send, one would define a callback hook for the xhr 
object, something like explained  
here: https://www.w3schools.com/xml/xml_http.asp

  xhr.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
  // do something with this.responseText;
  console.log('got response');
}
  };


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/cbfe7308-6176-4ffa-ba2c-238b686495fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] new release/branch of pyramid_formencode_classic

[Jonathan Vanasco]

i released the 0.2.x branch of `pyramid_formencode_classic` yesterday, and 
just updated it to `0.2.2` with some packaging fixes and built-in 
debugtoolbar support.

the package is an adaptation of the pylons' formencode integration into 
pyramid. it does not work exactly the same (no pylons style validation 
decorators), but it allows for existing formencode based pylons projects to 
be ported onto pyramid pretty quickly.

the 0.2.x branch has a streamlined api, test coverage, supports multiple 
forms on a page, and handles form reprints much better.  the package 
requires formencode-2.0.0a1 alpha (released in 2016 and rather stable); it 
is not compatible with formencode-1.3.

just to be clear: the target audience is for legacy pylons projects, 
already-converted pylons projects, and developers who need to leverage 
existing formencode work. *new pyramid users should still use Colander.*

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/2ebc2564-b847-4644-9141-b18760ef7ba4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] WebTest 2.0.30 has been released.

[Jonathan Vanasco]

clarification: oAuth2 supports some other flows/tokens too - but the Bearer 
concept is vert common.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/03798a95-8265-49d6-bad5-a30a274ca353%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] WebTest 2.0.30 has been released.

[Jonathan Vanasco]

On Saturday, June 23, 2018 at 2:26:55 PM UTC-4, Gael Pasgrimaud wrote:
>
> It's like basic auth but use tokens in the Authorization headers instead 
> of base64(user:pass). It's a bit more secure... JWT token may also 
> contains 
> some extra data (user infos, api scope, etc.) but those are encrypted 
> using 
> public/private keys. I don't know much about bearer.  
>

A "Bearer Token" is the common type of "Access Token" in oAuth2 and used by 
many APIs.  

In oAuth2, a client often authenticates with a key+secret and is issued a 
"Bearer Token" in return. The "Bearer Token" is then sent in the HTTPS 
headers, and can be invalidated from the authorizing application.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/fef1352a-3071-4a09-a919-331e393d52b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] pyramid_mailer and html email

[Jonathan Vanasco]

I've been lazy and just implemeted html email via pyramid_mailer as 
quoted-printable

email_html = Attachment(data=email_html, disposition='inline', 
transfer_encoding='quoted-printable')

i realized the pyramid_mailer encoders don't implement any of the email 
quirks for old RFCs and clients:

* lines about 76/78 chars long; never more than 998 chars long
* encode whitespace

has anyone not followed those rules and had deliverability issues?

does anyone have a simple tool/function for encoding a string to the older 
formats i can leverage?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9dadd696-e1da-4d56-8b2d-e8045071826f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] redis debugging?

[Jonathan Vanasco]

great!  i'll try to split some of my test stuff out and put it on github.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/18910aaf-0f6c-4eb3-8dd1-2a64a811b1f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] redis debugging?

[Jonathan Vanasco]

i've been toying with a debugtoolbar for redis connections.

the big issue is that python's redis doesn't have any sort of logging or 
callback facility

as a workaround, i've been using a proxy class as a logger... which 
requires manually wrapping/replacing the redis client objects.

because of that, i'm not inclined to build anything for a public release 
myself.  it's just all wonky stuff.

does anyone else have a need for this and interested in collaborating? if 
so, i'll try to pull stuff out to a public repo.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e74e7337-97c7-4fe6-8235-6402287fc5bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] WebOb 1.8.2 has been released

[Jonathan Vanasco]

On Wednesday, June 6, 2018 at 2:04:20 PM UTC-4, Bert JW Regeer wrote:
>
> Recently some changes were made to remove any last references to `key`, so 
> that things would error if you did try to use it, but those are on master 
> only and have not been back ported to 1.8-branch (yet, I may still do that. 
> I forgot to do it this go-around). 
>

Ah I thought that was in this release, sorry.  I was the one who found that 
issue in the docs code.  The point I wanted to get across with the above, 
is that if you're (erroneously) using `key` now, your code is already 
broken because it's been accepting that kwarg that has does nothing for 
years, without raising an exception.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/64926c81-9b7d-4c23-a8ff-f73a69b4846f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: WebOb 1.8.2 has been released

[Jonathan Vanasco]

I believe this is also the first release that no longer accepts the kwarg 
"key".  it's functionality was deprecated years ago, and it was officially 
removed a few years back too. a kwarg that does nothing was left in a few 
spots though.  if anyone experiences errors from this, their code was 
already broken for several years, but they didn't realize.

I'm just mentioning this on the off-chance anyone has an issue.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/fdf4ef03-7e5f-4061-a87d-53efadcd74e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: is it possible to execute code at a certain point in accessing a view?

[Jonathan Vanasco]

Thanks a ton, Bert!

This is exactly the sort of functionality I was hoping for.  I had seen the 
L114-L139 

 reference 
you mentioned, so was hoping there was another way.  Looks like all these 
new pyramid upgrades over the last few years are what I didn't know i 
needed.
` 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/dc70eb78-d69b-49f9-b698-426d99349d41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] is it possible to execute code at a certain point in accessing a view?

[Jonathan Vanasco]

I'm doing some housekeeping on an app that has been a bit too lax on 
keeping to it's own coding standards.  

There have been a bunch of updates over the past few years to the views 
systems, so I'm hoping something may work for our needs...

In a handful of sections, it utilize class based views that rely on 
inheritance for setup routines.

for example...


 class Foo(object):
   def __init__(self, request):
self.request = request
... common setup ...

   @view_config(route_name="bar",)
   def bar(self):
 pass

I was wondering if it is possible to hook into pyramid after the Foo() is 
instantiated, but before `Foo.bar` is called.

What I want to accomplish, in case someone has a better suggestion:

* The views i'm dealing with generally handle form processing on an API.  
* There are a handful of common setup and form validation routines that 
happen on these
* I'd like to define and trigger the common validation in a parent class, 
to ensure it runs. a handful of views were not calling the correct 
validation routines, because people make easy mistakes like that.
* I could integrate this processing into __init__, but error reporting 
would be much easier if it occurs after __init__, so I can utilize the 
class instance.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/ee79f36e-f7f1-4f75-abb3-d4e31ca879f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Suggested way to issue requests from inside a view callable

[Jonathan Vanasco]

On Tuesday, May 1, 2018 at 5:34:15 PM UTC-4, jens.t...@gmail.com wrote:
>
>  
>
Jonathan, funny you mention Celery. I have used it for a while but the 
> experience has been horrible—the thing is ridden with bugs and problems, 
> barely maintained, and the list of issues on Github grows daily. Which is 
> why I raised this discussion: 
> https://stackoverflow.com/questions/46517613/python-task-queue-alternatives-and-frameworks
>
> Curious though, I rolled the Celery task integration myself because I 
> didn’t find any specific module for Pyramid. Is there some explicit support 
> module out there?
>

There is a pyramid_celery integration package 
https://github.com/sontek/pyramid_celery

I ended up rolling my own as well though, because I didn't know about it at 
the time.

We also have multiple Pyramid and Twisted systems that communicate with 
Celery over Redis and Postgres, so needed to build some glue anyways.

I'm not particularly happy with Celery, but it works for our needs.  The 
biggest problem with it is the documentation - it's almost always out of 
date or wrong.  Once you accept that and just ignore it, defaulting to 
reading the sourcecode, it's much easier to use.  That being said, I'm much 
more likely do deploy a replacement service in Erlang or similar than 
switch to another Python system. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/066ab3ed-4831-4480-9c96-d525b46c6546%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] DB QueuePool limit overflow and Pyramid sub-requests

[Jonathan Vanasco]

On Tuesday, May 1, 2018 at 1:17:31 AM UTC-4, jens.t...@gmail.com wrote:
 

> I’ve followed the suggested cookie cutter code exactly: 
> https://github.com/Pylons/pyramid-cookiecutter-alchemy, which uses 
> pyramid_tm and adds the SQLA session to that transaction manager.
>
 
Are you passing in `*use_tweens=True*` ?

https://docs.pylonsproject.org/projects/pyramid/en/latest/api/request.html#pyramid.request.Request.invoke_subrequest
https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/subrequest.html?highlight=use_tweens#subrequests-with-tweens

If not, the behavior you show is expected.  `pyramid_tm` handles the 
transaction begin/commit via tweens.  If the subrequests don't use the 
tweens, they're part of your main transaction and don't close the 
dbconnection or return it to the pool.  So they will require 1 db 
connection for main + [1 db connections for each subrequest].

If you are not enabling tweens in the subrequest, then Pyramid doesn't work 
how I'd imagine it to.  But this behavior makes sense for pyramid_tm 
without tweens enabled.

Sidenote: I don't know if transaction/pyramid_tm can handle subrequests 
like this.  You might want the functionality that Michael Merickel 
suggested above, by copying over the main db session.

Like most others though, I avoid subrequests at all costs. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5d4743af-52c2-489d-8344-1f688a085cb2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Suggested way to issue requests from inside a view callable

[Jonathan Vanasco]

It depends on what you're communicating with, and why.

If your synchronous tasks are done "quickly" or must be "blocking" -- like 
doing some oAuth or hitting an external API that is guaranteed to return a 
request within a second or two, I would just use `requests` from within 
Pyramid.

If you're concerned with extended processing on your end, or not using 
systems that guarantee a response within a given amount of time... I would 
use Pyramid to trigger a Celery task, and then have the page reload every 5 
seconds to poll the Celery backend for status.

The reason for the latter usage pattern is that you will end up with a mix 
of browser timeouts / dropped connections AND tying up Pyramid workers 
while you wait on the upstream data.  Blocking while waiting on `requests` 
to process the response will end up creating a bottleneck.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0291364d-0822-4e1b-9d0e-26a232d16d42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] DB QueuePool limit overflow and Pyramid sub-requests

[Jonathan Vanasco]

On Saturday, April 28, 2018 at 5:45:23 AM UTC-4, jens.t...@gmail.com wrote:
>
> From within the view function (i.e. handling the incoming request) I issue 
> 5 subrequests one after the other. Doing so I noticed that the number of 
> subrequests was bound by the pool_size + max_overflow, hence my question 
> here and in the SQLA group. 
>

My confusion in your wording was that "It shouldn't work like that".  
Pyramid handles subrequests independently, so it will create a new session 
with a connection checked out from the pool, and then should close and 
return it at the end.  I thought there may be something wrong with the 
connection pool, perhaps during startup or perhaps by the subrequest 
functionality, but looking at the subrequest source it doesn't look to 
spawn a new thread.

How are you handling your session connection and cleanup?  Are you using 
pyramid_tm? If so, are you using the `use_tween` on the invoke_subrequest 
to properly close each connection in the pool?  If not, how are you 
cleaning up your connections?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/742c3ea1-0637-4e5c-a6e7-3d10a39b3bdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] DB QueuePool limit overflow and Pyramid sub-requests

[Jonathan Vanasco]

I'm confused by wording on this, or perhaps the functionality in Pyramid.

> When I set the pool_size 

 of 
the engine to *N* and max_overflow 

 to *M* then I can issue only a max of *N+M* subrequests, after which I get 
an exception:

Assuming you have 1 request and 5 subrequests, shouldn't there only be 2 
connections needed in the pool (i.e. the main request establishes a first 
connection, then subrequest 1 establishes a second connection which is 
re-used by 2-5)?  You wouldn't be able to save a connection like this if 
you had recursive subrequests - but that would be a design flaw in the 
application logic.

If you're connecting to sqlalchemy during your setup, you can screw up the 
connection pool unless you call `engine.dispose()` (see a thread from a few 
weeks ago), because SqlAlchemy's connections and pool aren't forksafe or 
threadsafe.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/ca945b9a-b9ce-4a2a-aede-f4a07226cb34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] API file upload

[Jonathan Vanasco]

On Wednesday, April 18, 2018 at 4:54:41 PM UTC-4, Bert JW Regeer wrote:
>
> Using request.body_file is actually better than using body_file_raw. The 
> raw reads directly from the underlying wsgi.input, which means if you are 
> using wsgiref or some other server that doesn't correctly input terminate 
> wsgi.input you may potentially deadlock reading forever because the 
> wsgi.input may be directly connected to the remote socket, and if you read 
> past CONTENT_LENGTH things go awry.
>

Wow. I did not know that. Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/b78adc93-11b5-4089-9ae8-a5a7f3aef028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: API file upload

[Jonathan Vanasco]

I was confused on this years ago. The problem is in naming...

`curl` is concerned with the HTTP method `POST`.  Let's call that `HTTP 
POST`.  If you look at the `curl` documents for the difference on the 
various `--data-` options.  Those options will `HTTP POST` data in 
different formats and encodings.

WebOb's `request.POST` isn't the same as `HTTP POST` though; it's a 
convenience method that pre-processing structured form data submissions via 
`HTTP POST` 
(see 
https://docs.pylonsproject.org/projects/webob/en/stable/api/request.html#webob.request.BaseRequest.POST)

If you look at the source of webob's POST 
https://github.com/Pylons/webob/blob/master/src/webob/request.py#L749-L797 
you'll see that it's trying to parse the raw data such as 
`self.body_file_raw` and `self.body_file` into form values.

Since your `HTTP POST` data isn't a form, you want to avoid all that 
processing and just operate on the raw data.  IIRC `body_file_raw` is 
better for uploads, because it avoids reading/consuming the body (which may 
be a stream instead of a static file).  I default to `body_file_raw` out of 
habit.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/3d8848d6-87cc-4fab-9e46-eede7457e878%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

On Monday, April 9, 2018 at 8:45:45 AM UTC-4, Tres Seaver wrote:
>
> > Somewhere after there, i believe.  It depends on the server.  uwsgi and 
> > unicorn do it in different places.  i think waitress essentially does it 
> > too. 
>
> Waitress is not a forking implementation:  it uses threads. 
>

Yes, Sorry, I should have been more clear. The SqlAlchemy 
session/pool/connection is not forksafe and not threadsafe. For the purpose 
of dealing with this type of situation (database connection in main 
routine), forked and threaded workers are affected similarly and 
essentially the same.  I was hoping to convey that switching from 
gunicorn/uwsgi to the waitress won't necessarily address this – one needs 
to `dispose()` of the pool.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/b1f6936d-cf40-4d8f-98e6-d154c8a2e7fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Linode sponsors Pylons Project web hosting for 3 years

[Jonathan Vanasco]

I've been using linode for several years. They're great.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/c9d8ab37-b764-4d2b-8a4a-0470cdf7f4bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

I confirmed with mike @ sqlalchemy.  calling `dispose()` will not 
explicitly `close()` connections that are already checked out. It just gets 
rid of the the connection pool instance, so you have fresh connections in 
the child processes-- the pre-fork connections themselves will be garbage 
collected out.  You shouldn't need to call "close()" even through the 
connection will still be open in your scope.




-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/62e2418c-5cfe-49c1-ad97-885d6e1e15cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

On Monday, April 2, 2018 at 12:09:26 PM UTC-4, Zsolt Ero wrote:
 

> So if I understand right, the engine in a gunicorn / process worker 
> Pyramid app is global / shared between all processes, right? And at 
> the time of forking, there should be no dbsessions being active and no 
> connections in the connection pool, right? 
>

Sort of. I'll try to explain this simply and using terms closer to you. The 
engine is global but you DON'T want to share it between processes because 
of Python's copy-on-write behavior and how connections are handled as 
file-descriptors.  When each process creates a new connection (or otherwise 
updates the pool), the pool is basically copied into a per-worker variable 
then updated.  If you have active connections in the pool *before* the 
fork, everyone gets a copy of that connection and thinks it is theirs -- 
but if the pool is empty, everyone gets an empty pool.
 
 

> Where is the forking happening in a normal Pyramid app? After the main 
> function finishes / "return config.make_wsgi_app()"? 
>

Somewhere after there, i believe.  It depends on the server.  uwsgi and 
unicorn do it in different places.  i think waitress essentially does it 
too.  gunicorn offers a post_fork hook 
(http://docs.gunicorn.org/en/stable/settings.html) which you can use to 
trigger a dispose.   

If you connect within the __main__ to pull config settings out of the db, 
you generally want to immediately drop the connection.  Doing it within 
your startup routine and before the fork is fine - but you can also do it 
after the fork.  All that matters is that the engine is disposed before you 
use it.

I'm checking with SqlAlchemy, but it looks like `dispose` doesn't 
explicitly close the session - it just removes it from the connection pool 
which will effectively close it during garbage collection.

calling close()+dispose() in your startup should be fine.  

there's another technique to protect against forks that mike (bayer of 
sqlalchemy) has been using in openstack and may bring to the main project, 
but I'm not sure if/when that will happen.

if you think you may jump between platforms and use multiple 
libraries/patterns that aren't fork-safe, I wrote a library that turns 
uwsgi & gunicorn events into subscribers 
(https://github.com/jvanasco/pyramid_forksafe) that make the cleanup work a 
little more portable.  We use sqlalchemy, pymongo and pycrypto which are 
all not forksafe.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/82a9fae4-2fe5-452e-8388-68007f8c2c6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

On Monday, April 2, 2018 at 12:07:14 PM UTC-4, Bert JW Regeer wrote:
>
>
> This is only required if you are not using pyramid_tm. If you are using 
> pyramid_tm which is what the sqlalchemy cookie cutter does, you do NOT need 
> to add this.
>

Thanks, Bert.  I stand corrected.  pyarmid_tm has a finish() that 
eventually calls a session.close() in Zope.SqlAlchemy.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e1f81475-8b31-44ca-b0cf-af3448a3b131%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

On Sunday, April 1, 2018 at 8:18:39 PM UTC-4, Zsolt Ero wrote:
>
> Hi Jonathan, 
>
> I'm not 100% sure I understand when are you talking about the 
> "standard" Pyramid way of the cookiecutter template's get_tm_session's 
> implementation and when are you talking about my special addition of 
> using the db to set up the app.
>

I don't know why it was removed from the cookiecutter.

It's been present in the cookbook and docs for years, see the cookbook:

https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/database/sqlalchemy.html

I've opened a ticket on the cookiecutter to correct this.


I don't know how and when is a gunicorn process fork happening in the 
> standard Pyramid app, but I presume it's tested and proven by people 
> who understand the internals of SQLAlchemy connections. By this I mean 
> that I think about request.dbsession as something which is managed for 
> me. This results in 1 idle connection per gunicorn worker. 
>

It's not.  SqlAlchemy, Pyramid and Zope.SqlAlchemy (the transaction 
extension) do nothing to check or manage this.  Anything that is happening 
is pure coincidence and luck.  

If you connect to the database before the fork, you need to call 
`engine.dispose()`  If you don't connect to the database before the fork, 
you don't need to call `dispose`. 


So far, my workaround which seems to work is the following: 
>
> dbsession = config.registry['dbsession_factory']() 
> siteconfig = get_siteconfig(dbsession) 
> dbsession.close() 
> del dbsession 
>
> (I'm deleting it to make sure that I cannot accidentally call it again). 
>
> I tried calling dbsession.connection().engine.dispose() but it didn't 
> do anything. 
>
> dbsession.connection().close() seems to behave the same as if I call 
> dbsession.close(). 
>

I can try to spin up your test later.  `dispose` is really what you need to 
call in your situation.  `close` just 'ends' the session and returns the 
connection to the pool; `dispose` will close the actual db connection. 
(perhaps this is being done by the delete, i don't know).   it is important 
to close the connection and reset the pool before the fork, because you run 
the risk of different workers using the same connection.


 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e5c2df72-f9b8-4de9-92e1-639e13e52391%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Jonathan Vanasco]

Pyramid isn't opening the database connection, the SqlAlchemy pool is.

On first glance, you're connecting to the database in your main(). This is 
guaranteed to create issues with SqlAlchemy's connection pool when you 
involve multiple workers/threads/processes.  In some situations the 
connection gets reused by multiple processes, in others it gets lost.

The proper way to fix this is to call `engine.dispose()` 
[http://docs.sqlalchemy.org/en/latest/core/connections.html#engine-disposal] 
either immediately after you use it, or in a post-fork hook.  

I also don't see an add_finished_callback registered to call 
'session.close() on every request (or at least the ones where you grab a 
dbsession), or the close() being called in any other way (like a tween).  

Between both of those traits, you're going to have a connection pool that 
constantly needs new connections (it has to wait for python's GC to clear 
out the connection), and might re-use existing connections unreliably.

There could be other factors causing this too, but you should integrate a 
`dispose` after the app starts and add a utility to register a 
session.close() via add_finished_callback().  If that doesn't solve it, 
there are a lot of docs in SqlAlchemy on the connection pool specifics.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/d21f0d7e-f20c-4760-9e03-9e057dde376c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Help converting Pylons {name}{.format} style routes to Pyramid

[Jonathan Vanasco]

There's also some pyramid docs on predicates and route factory under URL 
dispatch.   I think that might be what you need - there are some threads in 
this group on those topics.

I also wrote a library `pyramid_route_7` 
(https://github.com/jvanasco/pyramid_route_7) that extends the config to 
allow macros.  Instead of adding a route with `add_route` you use 
`add_route_7`, which then expands the macro and calls `add_route` behind 
the scenes.  That technique might help you a bit too.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/acf7087c-0f3b-464b-91f1-ed6df5d6f4af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Re: CSRF token implementation

[Jonathan Vanasco]

you're absolutely correct.   i used a very bad choice of words and should 
have been specific because I was thinking of something weird. i meant to 
refer to using the new storage policy to implement the "encrypted token 
pattern", which basically bootstraps a micro session into a first dedicated 
csrf cookie, then programmatically constructing a request with a second 
cookie set by the client.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/49f94c31-af2a-43ba-8305-cf75f3092b36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: CSRF token implementation

[Jonathan Vanasco]

There's not enough here for me to guess why, but I wanted to note in 
Pyramid 1.9.x you can store the CSRF in a cookie (instead of in the 
session).  It may be worth upgrading to use the new storage policy (and 
compare two cookies) before fixing this.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a0ed35da-c332-4108-a277-de668bd54e46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] is there an easy way to handle a unique http and https CSRF tokens ?

[Jonathan Vanasco]

I ended up writing a small utility with debugtoolbar support for 
this https://github.com/jvanasco/pyramid_csrf_multi_scheme

I'm not entirely sure it's needed, but leaned towards "better safe than 
sorry". 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/eb895435-bf29-479f-ae44-af1bd1f208f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] is there an easy way to handle a unique http and https CSRF tokens ?

[Jonathan Vanasco]

On Monday, December 4, 2017 at 3:31:19 PM UTC-5, Michael Merickel wrote:
>
> > I'm in the process of migrating to the new cookie based csrf policy, 
> and it doesn't seem like there is an easy way to run different tokens on 
> HTTP vs HTTPS short of writing a new ICSRFStoragePolicy utility and 
> plugin.  Has anyone worked on this before?
>
> Specialized use-cases are the reason we added the ability to provide a 
> custom storage policy. You'll probably want to write one.
>


Thanks, Michael! I didn't really think this was that specialized, and 
wanted to ensure I wasn't missing anything obvious.  

Building a custom policy is pretty easy - it only took about 30 minutes to 
prototype with unit tests.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/9ec87cfb-ca98-41b8-a632-884c84e19e0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] is there an easy way to handle a unique http and https CSRF tokens ?

[Jonathan Vanasco]

We strive for 100% HTTPS, but need to support some HTTP endpoints.  Those 
endpoints have CSRF needs.

I'm in the process of migrating to the new cookie based csrf policy, and it 
doesn't seem like there is an easy way to run different tokens on HTTP vs 
HTTPS short of writing a new ICSRFStoragePolicy utility and plugin.  Has 
anyone worked on this before?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0e098559-5bdc-4e71-8f23-a8ab891f2957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: handling multiple "sessions"

[Jonathan Vanasco]

I cobbled together a first version this afternoon 
- https://github.com/jvanasco/pyramid_session_multi

if anyone has suggestions, please share.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e0071d15-720d-4ecb-976b-573215980749%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] handling multiple "sessions"

[Jonathan Vanasco]

Has anyone done work on this and can offer any learnings/gotchas?

I need to handle multiple "session" containers each request object.  One of 
our products is a platform with whitelabel functionality, so a request can 
happen on the platform domain OR a partner domain, and can be stored 
clientside or serverside:

* platform domain - serverside container (https only)
* platform domain - clientside container (http+https)
* partner domain - clientside container (http+https)

The current approach works, but is not nice to maintain or grow. We have a 
new ISessionFactory subclass ISessionHTTPSFactory which is registered to 
`request.session_https`.  A custom session factory is then used to handle 
the right http/https backend stores, and some request methods are used to 
streamline accessing them.

What I'd like to move towards, is dropping the ISessionHTTPSFactory subclass, 
and either use only `config.set_request_property` and/or having the  
ISessionFactory 
subclasses automatically created/managed as needed.  The goal is to not 
require creating/managing these subclasses and their mountpoints hardcoded, 
but allow them to be easily changed.  I need to add two more clientside 
"sessions" for https connections to our platform, and the current method 
won't scale like that.  

These are all true "sessions" and leveraging pyramid ISession libraries but 
the data has different retention needs and usage policies.  we often split 
things into multiple apps, so a few might have access to every payload but 
others can only access one payload.

If anyone has done work, or even thought on this, i'd be grateful to know 
any tips or missteps.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/1cc768d9-dabc-4b86-8ce9-d7dc7fe5f544%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: How to handle DB transaction failures/request race conditions

[Jonathan Vanasco]

The sqlalchemy cookie cutter uses pyramid_tm to handle the transaction. 
 The docs on that package should answer all your questions:

   https://docs.pylonsproject.org/projects/pyramid_tm/en/latest/

Basically what happens is this:

* the actual transaction management is provided by zope's transaction 
package (http://zodb.readthedocs.io/en/latest/transactions.html)
* the glue between zope and sqlalchemy is zope.sqlalchemy 
(https://pypi.python.org/pypi/zope.sqlalchemy)
* the template uses pyramid_tm to register a transaction around the request 
using a pyramid tween.
* pyramid_tm manages a "meta transaction" that has different components: 
one of which is the database "transaction", but there might also be joined 
transactions for mailing, celery, and other tasks.

When the database transaction fails, an exception is raised.  You can write 
a view to catch that exception, as explained in the pyramid_tm docs.

In your example, two things can happen based on the The order of access in 
the race condition and your DB's configuration.

1. One transaction succeeds, the other fails.
2. Both fail (deadlocks or other db errors).

SqlAlchemy should trap almost any driver error regarding this, and abstract 
it into one of it's own core exceptions 
(http://docs.sqlalchemy.org/en/latest/core/exceptions.html). 

When one of the joined transactions fails, the transaction manager notifies 
the rest to issue a rollback-- so the database should rollback, email 
queues are removed, etc.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/012c9079-61ed-4462-8d4f-a0a1b8ae4e33%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Recommended way to modify view return value

[Jonathan Vanasco]

>
> I considered building a custom renderer, but I can’t figure how I’d pass 
> the schema into it. 

I could also just use a vanilla decorator, but that covers up the view call 
> signature.


I do similar stuff in a variety of ways.

I almost-always augment the request object with an `add_request_method` 
that sets/stores/calculates the needed info for the active route.  On some 
routes, that data is preloaded using a decorator, on others it is looked up 
as needed by calculating info from the request's matched_route.

The second value to `render` methods of factory instances is `system` which 
should have the `request` object in it (reference the pyramid source or the 
cookbook for more info on that).  So the request, and custom method, is 
easily available within the custom renderer.

The big reason why I prefer using `add_request_method` to stash data like 
this is for ease in debugging.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/8e88d77f-1f80-41ff-847c-fdc1bd99ffec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid_tm reason for ROLLBACK

[Jonathan Vanasco]

The error is most likely in your Pyramid code.  If SqlAlchemy failed, there 
would be a SqlAlchemy error logged.  Your example shows SqlAlchemy 
completing an INSERT, but then being instructed to ROLLBACK by the 
transaction manager.  You should increase your Pyramid logging levels.

Between your code example and error patterns, I would guess the exception 
is being raised in the JSON renderer.  Most likely that is because you're 
using the stock JSON renderer, which doesn't support most common Python and 
SqlAlchemy datatypes (such as datetime, decimals, etc).

I suggest you look more closely at your exception logging, and run some 
tests to see if the active JSON encoder can handle your datatypes. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/3b84e66e-c6f9-44b8-aa21-6509616821ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Distributing a REST API Pyramid project

[Jonathan Vanasco]

originally we did your approach, but now we use multiple discrete packages, 
and manage the virtualenv with fabric

|python_packages/
| |app_pyramid/
| |  |---setup.py
| |  |---app_pyramid/
| |app_core/
| |  |---setup.py
| |  |---app_core/
| |app_celery/
| |  |---setup.py
| |  |---app_celery/
| |app_model/
| |  |---setup.py
| |  |---app_model/
|fabfile.py

the fabric file is used to manage everything and proxy commands to 
setup.py/pip/etc. it can build packages as needed, install from source, or 
even deploy into select environments.


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/24488b66-9416-4d95-9bd0-e14096af63e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Porting a ZOPE (2.13) application into PYRAMID

[Jonathan Vanasco]

+1 to the advice above on a gradual migration via failovers or  "strangler"

One method that once worked well for me in the past: viewing everything as 
a Service Oriented Architecture and extending the old system with new 
routes that enabled it to be the Auth/Login component for the new system. 
That let the old system act as-is with no change.  When everything got 
migrated over to the new system, the auth endpoints were switched over to 
the new system too.

An example flow:

* User visits NEW page.  NEW redirects to OLD for auth.
* User logs in on OLD, which now has an auth callback info (e.g. oauth).
* * OLD handles login and sets it's own cookies/session.
* * OLD redirects to NEW auth-in endpoint
* NEW sets it's own sessions/info based on the auth-in token and background 
data exchange.





-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/cff20da8-82c4-473d-9c9b-d55026f7357c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] has anyone made a tween/middleware/etc to adapt status codes depending on http protocol?

[Jonathan Vanasco]

Hoping to not reinvent the wheel if possible... 

I'd like to write our code redirects as HTTP/1.1 when applicable (such as 
303SeeOther on a successful post submission), but downgrade it to 
HTTP/1.0's 302 redirect if the client hasn't identified HTTP/1.1.  

Has anyone worked on this?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6b46db55-3a86-4bca-8460-26d6e7a62acd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: Recommendation for a Python-based wiki

[Jonathan Vanasco]

A lot of the hosted commercial services are freemium.  i think pbwiki gives 
you a handful of users on their free tier -- all you really need are 2 
users (one set of credentials for you, another one for your clients)

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e770a025-23c0-4871-a138-22cdf8e2eeec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] odd edge-case that I can't re-create

[Jonathan Vanasco]

one of our login forms had an error on it. we log python exceptions to sql 
in a tween (pretty much using the same logic as in pyramid_exclog) under 
pyramid 1.8.3

here's the edge case:

the broken form got hit by a botnet, and the nonstop errors ate up all the 
space on the dedicated postgreql server logging device.  the sqlalchemy 
connection was erroneously not setup for autocommit, and that wedged the 
underlying db connection into a broken transaction that never cleaned up. 
 so even after space was restored on that device, all exception logging was 
broken because the finished callback never fired.

the tween logic was pretty much something like this:

 try:
  return handler(request)
 except Exception as e:
  # record e to db
  raise
   
somehow, a stale database connection was created within the Exception block 
and never released.

i've tried for hours, but I can't seem to create a use case situation where 
I can break a connection in the `try` or `except` blocks, and it doesn't 
get cleaned up by the finished callback under 1.8.3 or 1.9.x.  

does anyone know what might cause a finished callback to *not* execute?

i've mostly fixed this by moving to an autocommit, but i'd like to figure 
out what the heck happened.  

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0ac394f7-5c7f-4e00-a5b9-7fc50baed6d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Minor encoding problem

[Jonathan Vanasco]

On Thursday, July 6, 2017 at 4:56:13 AM UTC-4, Steve Piercy wrote:
>
> I would suspect one of your Apache's not setting the character set to 
> UTF-8, but something less robust or not at all.  


I read the question above wrong, and thought the issue was in reading a 
file - not serving it.  Steve is right, this is most-likely the reason.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a1e916f9-18b3-4689-a9a3-58c41a7aa2ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.