Re: [pylons-discuss] Problems when using Gunicorn and Postgresql together

[Arndt Droullier]

I use a standalone connection pooling for Postgresql in some cases. It is
faster and more efficient than
doing full connects to Postgresql directly in each request.

https://www.pgbouncer.org/

Arndt.

Am Di., 21. Nov. 2023 um 16:14 Uhr schrieb Eldav :

> Thank you Jonathan,
>
> after asking my question, I did more googling and found this :
>
> https://docs.sqlalchemy.org/en/20/core/pooling.html#pooling-multiprocessing
>
> It does mention Engine.dispose :) I tried their solution #4, which seemed
> to be the one which fit best in my code. But somehow I feel it isn't quite
> satisfactory, I see a lot of connects / checkout, which seems inelegant and
> maybe bad for performance.
>
> Laurent.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/e22c60a5-978f-4204-81a6-c2c64b26475bn%40googlegroups.com
> 
> .
>


-- 
--
nive.co 
Arndt Droullier
Tel 0221-2725782
Nive GmbH, Händelstr. 26, Köln, Germany

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAJYR-0NPoXCDu_oTG%3DLNaugF9uunUwGXdRswSCz_X0XvhkG%3D_w%40mail.gmail.com.

Re: [pylons-discuss] Problems when using Gunicorn and Postgresql together

[Eldav]

This should not happen. Do you know which cookiecutter you used, and when?  
This should not happen in the most recent cookiecutter.


As far as I remember, I used the official cookiecutter, but that was a few 
years ago (around the time when Pyramid 2.0 was released, and I felt the 
need to synchronize my code with it.

Based on what you're saying, I should probably do it again now, as a first 
step :) I'll let you know how it went.

Thank you again,

Laurent.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5091b343-c00f-4d3d-b3d0-e2cc38eeb5cdn%40googlegroups.com.

Re: [pylons-discuss] Problems when using Gunicorn and Postgresql together

[Eldav]

Thank you Jonathan, 

after asking my question, I did more googling and found this :

https://docs.sqlalchemy.org/en/20/core/pooling.html#pooling-multiprocessing

It does mention Engine.dispose :) I tried their solution #4, which seemed 
to be the one which fit best in my code. But somehow I feel it isn't quite 
satisfactory, I see a lot of connects / checkout, which seems inelegant and 
maybe bad for performance.

Laurent.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e22c60a5-978f-4204-81a6-c2c64b26475bn%40googlegroups.com.

Re: [pylons-discuss] Problems when using Gunicorn and Postgresql together

[Jonathan Vanasco]

> Namely, if you deploy with Gunicorn a Pyramid + PostgreSQL app based on 
the standard cookiecutter, you will run into problems, because the 
connection to the DB can't be shared between the processes, so each process 
needs to have its own connection to the DB.

I forgot to mention...

This should not happen. Do you know which cookiecutter you used, and when?  
This should not happen in the most recent cookiecutter.

The cookiecutter works by adding a transaction/thread safe session to each 
request: 
https://github.com/Pylons/pyramid-cookiecutter-starter/blob/latest/%7B%7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/sqlalchemy_models/__init__.py#L87-L127

It also requires pyramid_tm, which will cleanup the connections at the end 
of the request lifecycle.  If you decide to not use pyramid_tm, then you'll 
have to use a cleanup routine like this one when you grab the request.  
This was one the pattern in the pyramid cookiecutter, but it was taken out 
because pyramid_tm effectively does that itself.

  
  
https://github.com/jvanasco/pyramid_oauthlib_lowlevel/blob/main/tests/oauth2_app/model/__init__.py#L71-L77

The only way the connection pool sharing should be able to happen is if you 
grabbed a database connection before the fork - like during application 
setup, which would then require a call to Engine.dispose.

If you are not consciously grabbing a database connection before the fork, 
you should do a code audit to figure out where the first connection is 
being made and how it is recycled, as the last few versions of the 
cookiecutter do defend against this behavior.




On Tuesday, November 21, 2023 at 1:06:43 AM UTC-5 Mike Orr wrote:

> On Mon, Nov 20, 2023 at 4:14 PM Jonathan Vanasco  
> wrote:
> >
> > SQLAlchemy supports this via `Engine.dispose()`, which is the documented 
> way of handling a post-fork connection:
> >
> > https://docs.sqlalchemy.org/en/13/core/connections.html#engine-disposal
>
> Yes, that sounds familiar.
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/984f5ea0-5d10-416b-83f0-c6d19d30ad9dn%40googlegroups.com.