Re: Database pooling vs. persistent connections

On Feb 28, 2013, at 1:43 PM, David Cramer wrote: > It is most definitely not an "error" to have less connections available than > workers, considering workers may serve different types of requests, and will > now persist the database connection even after that request has finished. If you have

Re: Database pooling vs. persistent connections

It is most definitely not an "error" to have less connections available than workers, considering workers may serve different types of requests, and will now persist the database connection even after that request has finished. It may not be "the most correct", but a concurrent connection is ve

Re: Database pooling vs. persistent connections

On Thu, Feb 28, 2013 at 4:10 PM, Christophe Pettus wrote: > > On Feb 28, 2013, at 11:09 AM, David Cramer wrote: > > > Immediately for anyone who has configured more workers than they have > Postgres connections (which I can only imagine is common among people who > havent setup infrastructure lik

Re: Database pooling vs. persistent connections

On 28 févr. 2013, at 22:05, Christophe Pettus wrote: > It would be helpful to have a backend method that performers the "restore > connection between uses" function, rather than just use connection.abort() > (of course, the default implementation can use that). For example, on > PostgreSQL, A

Re: Database pooling vs. persistent connections

On Feb 28, 2013, at 11:09 AM, David Cramer wrote: > Immediately for anyone who has configured more workers than they have > Postgres connections (which I can only imagine is common among people who > havent setup infrastructure like pgbouncer) things will start blowing up. If they have this co

Re: Database pooling vs. persistent connections

One comment on the patch (which I generally approve of entirely): It would be helpful to have a backend method that performers the "restore connection between uses" function, rather than just use connection.abort() (of course, the default implementation can use that). For example, on PostgreSQL

Re: Database pooling vs. persistent connections

On 28 févr. 2013, at 20:09, David Cramer wrote: > Can we please change this so it defaults to off, and just document how to > turn it on and in what situations you should turn it on? > > In my opinion this default-on feature caters to a very specific audience, and > will cause a lot of unexpe

Re: Database pooling vs. persistent connections

Can we please change this so it defaults to off, and just document how to turn it on and in what situations you should turn it on? In my opinion this default-on feature caters to a very specific audience, and will cause a lot of unexpected behavior with other users. Here is the tl;dr of an argu

Re: Database pooling vs. persistent connections

On 22 helmi, 11:21, Aymeric Augustin wrote: > > Second -- Tasks out of requests. Core developers participating in this > > discussion already have this in mind, Aymeric has commented about it in > >https://code.djangoproject.com/ticket/17887, but it was left out of the > > discussion on the thread

Re: Database pooling vs. persistent connections

Hi Shai, Thanks for compiling this list ! My answers below. > First -- the very mention of this topic reminded me of > https://code.djangoproject.com/ticket/9964, which was my pet bug for about > two > years. The point of that bug, though, was to make sure that transactions are > properly clo

Re: Database pooling vs. persistent connections

Hi, On Sunday 17 February 2013, Aymeric Augustin wrote: > **tl;dr** I believe that persistent database connections are a good idea. > Please prove me wrong :) > > [...] > > So -- did I miss something? I think you haven't -- but several points came to mind, which weren't discussed in this threa

Re: Database pooling vs. persistent connections

I don't really see what point such an option would serve. This is *not a connection pool*. Currently Django uses one open connection per thread, this simple keeps them open between requests. That means before: you needed O(concurrent threads) connections, now you need O(total active threads) connec

Re: Database pooling vs. persistent connections

One question: does this proposal include some way of specifying a maxiumum number of outstanding connections to the database from a single process? Looked but didn't see it in the pull request. In my experience, most PostgreSQL instances don't have a whole lot of breathing room in terms of th

Re: Database pooling vs. persistent connections

On 02/19/2013 03:04 AM, Anssi Kääriäinen wrote: > I hope this discussion is about what to do at request finish/start > time. > > I am very strongly opposed to anything where Django suddenly changes > connections underneath you. At request finish/start this is OK (you > should expect new connection

Re: Database pooling vs. persistent connections

On 19 helmi, 12:20, Aymeric Augustin wrote: > On 19 févr. 2013, at 11:04, Anssi Kääriäinen wrote: > > > Maybe we need another setting for what to do in request.start. It does > > seem somewhat likely that users could do SET SEARCH_PATH in middleware > > to support multitenant setups for example,

Re: Database pooling vs. persistent connections

On 19 févr. 2013, at 11:04, Anssi Kääriäinen wrote: > Maybe we need another setting for what to do in request.start. It does > seem somewhat likely that users could do SET SEARCH_PATH in middleware > to support multitenant setups for example, and expect that set to go > away when connection is cl

Re: Database pooling vs. persistent connections

On 19 helmi, 02:31, Carl Meyer wrote: > On 02/18/2013 02:27 PM, Aymeric Augustin wrote: > > > Problem #1: Is it worth re-executing the connection setup at the beginning > > of > > every request? > > > The connection setup varies widely between backends: > > - SQLite: none > > - > > PostgreSQL:ht

Re: Database pooling vs. persistent connections

Just FYI, in the DB2 backend for Django, we have exposed a custom database property PCONNECT and have defaulted it to true for a long time now. What this does is that it uses persistent connections that is supported by the DB2 Python driver (http://code.google.com/p/ibm-db/ wiki/APIs#ibm_db.pconnec

Re: Database pooling vs. persistent connections

On 02/18/2013 02:27 PM, Aymeric Augustin wrote: > Problem #1: Is it worth re-executing the connection setup at the beginning of > every request? > > The connection setup varies widely between backends: > - SQLite: none > - PostgreSQL: > https://github.com/django/django/blob/master/django/db/backe

Re: Database pooling vs. persistent connections

On 18 févr. 2013, at 23:30, Karen Tracey wrote: > Will persistent connections be able to ensure that "bad connections" (e.g. > those in state "current transaction is aborted, commands ignored until end of > transaction block") don't leak from one request to another? I'm calling connection.abor

Re: Database pooling vs. persistent connections

On Mon, Feb 18, 2013 at 5:33 PM, Alex Gaynor wrote: > As far as I know, the MySQL timeout time is configurable, and the default > is 8 hours. > Assuming this is what does: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_wait_timeoutthat sounds right. I guess I have jus

Re: Database pooling vs. persistent connections

As far as I know, the MySQL timeout time is configurable, and the default is 8 hours. Alex On Mon, Feb 18, 2013 at 2:30 PM, Karen Tracey wrote: > Just a couple of random quick thoughts: > > Will persistent connections be able to ensure that "bad connections" (e.g. > those in state "current tra

Re: Database pooling vs. persistent connections

Just a couple of random quick thoughts: Will persistent connections be able to ensure that "bad connections" (e.g. those in state "current transaction is aborted, commands ignored until end of transaction block") don't leak from one request to another? MySQL, at least, will close a connection fro

Re: Database pooling vs. persistent connections

On 18 févr. 2013, at 14:47, Jacob Kaplan-Moss wrote: > This fix represents a cheap and easy performance improvement. Let's get it in. After working on a few iterations of the patch, I have identified two problems worth discussing. Problem #1: Is it worth re-executing the connection setup at th

Re: Database pooling vs. persistent connections

On 18 févr. 2013, at 19:51, Michael wrote: > I was just alerted to this by the reopening of ticket #18135. My initial analysis is incomplete. I'll comment on the ticket in a few minutes. > I will argue, however, that the AWS use case is a reason we need this to be > an OPTION, not automatic.

Re: Database pooling vs. persistent connections

On Mon, Feb 18, 2013 at 2:21 PM, Carl Meyer wrote: > On 02/18/2013 12:17 PM, Michael wrote: > > On Mon, Feb 18, 2013 at 2:03 PM, Jacob Kaplan-Moss > > wrote: > > > > On Mon, Feb 18, 2013 at 1:51 PM, Michael > > wrote: > > > I w

Re: Database pooling vs. persistent connections

On 02/18/2013 12:17 PM, Michael wrote: > On Mon, Feb 18, 2013 at 2:03 PM, Jacob Kaplan-Moss > wrote: > > On Mon, Feb 18, 2013 at 1:51 PM, Michael > wrote: > > I will argue, however, that the AWS use case is a reason we need >

Re: Database pooling vs. persistent connections

On Mon, Feb 18, 2013 at 2:03 PM, Jacob Kaplan-Moss wrote: > On Mon, Feb 18, 2013 at 1:51 PM, Michael wrote: > > I will argue, however, that the AWS use case is a reason we need this to > be > > an OPTION, not automatic. > > Check out Aymeric's patch; it is indeed an option triggered by > DATABASE

Re: Database pooling vs. persistent connections

On Mon, Feb 18, 2013 at 1:51 PM, Michael wrote: > I will argue, however, that the AWS use case is a reason we need this to be > an OPTION, not automatic. Check out Aymeric's patch; it is indeed an option triggered by DATABASES[CONN_MAX_AGE]. Jacob -- You received this message because you are s

Re: Database pooling vs. persistent connections

I was just alerted to this by the reopening of ticket #18135. I agree the routing on AWS is slow and represents a large amount of connections. This is something that AWS needs to address. I will argue, however, that the AWS use case is a reason we need this to be an OPTION, not automatic. Genera

Re: Database pooling vs. persistent connections

On 02/18/2013 06:47 AM, Jacob Kaplan-Moss wrote: > On Mon, Feb 18, 2013 at 7:41 AM, Aymeric Augustin > wrote: >> The goal of my proposal is to save the cost (in response time) of >> establishing >> the database connection, in vanilla Django, without compromising reliability. >> I'm trying to help

Re: Database pooling vs. persistent connections

Also, https://github.com/niwibe/djorm-ext-pool is based initially on postgresspool but works with sqlite and mysql. The only difference is that uses monky patching instead a separate backend. We also use it in production on a project a while ago and has not given us any problems. Andrey 2013/2/

Re: Database pooling vs. persistent connections

On Mon, Feb 18, 2013 at 7:41 AM, Aymeric Augustin wrote: > The goal of my proposal is to save the cost (in response time) of establishing > the database connection, in vanilla Django, without compromising reliability. > I'm trying to help people who aren't even aware of the problem :) > > The impl

Re: Database pooling vs. persistent connections

Le 17 févr. 2013 à 19:31, Anssi Kääriäinen a écrit : > I am not yet convinced that poolers implemented inside Django core are > necessary. In case I wasn't clear enough — I agree that Django core shouldn't provide connection pooling, because it'll be simplistic and hamper innovation outside of D

Re: Database pooling vs. persistent connections

+1 to django-postgrespool it works well for me. On Sunday, 17 February 2013 22:17:09 UTC, Carl Meyer wrote: > > On 02/17/2013 11:31 AM, Anssi Kääriäinen wrote: > > It seems SQLAlchemy has a mature pooling implementation. So, yet > > another approach is to see if SQLAlchemy's pooling implementati

Re: Database pooling vs. persistent connections

On 02/17/2013 11:31 AM, Anssi Kääriäinen wrote: > It seems SQLAlchemy has a mature pooling implementation. So, yet > another approach is to see if SQLAlchemy's pooling implementation > could be reused. (Maybe in conjunction with the above 'POOLER' idea). There is in fact an implementation of reusi

Re: Database pooling vs. persistent connections

On 17 helmi, 13:24, Aymeric Augustin wrote: > **tl;dr** I believe that persistent database connections are a good idea. > Please prove me wrong :) > > > > Since I didn't know why the idea of adding a connection pooler to Django was > rejected, I did some research before replyi

Database pooling vs. persistent connections

**tl;dr** I believe that persistent database connections are a good idea. Please prove me wrong :) Since I didn't know why the idea of adding a connection pooler to Django was rejected, I did some research before replying to the cx_Oracle SessionPooling thread. The best expl