Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-07 Thread Sunil
On Saturday 07 December 2013 06:32 AM, Michael Bayer wrote: [...] OK more or less that, please review my commit at https://github.com/zzzeek/sqlalchemy/commit/d1cc78479d988bd9acbcf395483d2130b0873b1c which moves handling of “overflow” into _inc_overflow() and _dec_overflow() methods - the

[sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Sunil Adapa
Hello, I have faced a problem in my production server (gevent based); when a connection attempt is made and MySQL server does not respond (due to listen backlog full), the whole application hangs. This seems to be because SQLAlchemy QueuePool does not allow multiple connection attempts

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Michael Bayer
On Dec 6, 2013, at 10:16 AM, Sunil Adapa su...@innopark.in wrote: Hello, I have faced a problem in my production server (gevent based); when a connection attempt is made and MySQL server does not respond (due to listen backlog full), the whole application hangs. This seems to be because

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Michael Bayer
On Dec 6, 2013, at 11:15 AM, Michael Bayer mike...@zzzcomputing.com wrote: Since the overflow lock seems to be to only maintain overflow count, I suggest that we increment the counter *before* connection attempt, don't hold the lock during connection attempt and then decrement the counter

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Michael Bayer
yeah, sorry I get this entirely now, ticket is http://www.sqlalchemy.org/trac/ticket/2880 and here’s a test, has nothing to do with gevent. the patch should be fine and I’ll see if i can get it into 0.8.4 as well. from sqlalchemy.pool import QueuePool from sqlalchemy.testing.mock import Mock,

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Michael Bayer
On Dec 6, 2013, at 6:01 PM, Claudio Freire klaussfre...@gmail.com wrote: On Fri, Dec 6, 2013 at 3:50 PM, Michael Bayer mike...@zzzcomputing.com wrote: +except: +self._overflow -= 1 +raise That also needs to aquire the lock. I think I’m

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Michael Bayer
On Dec 6, 2013, at 6:27 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Dec 6, 2013, at 6:01 PM, Claudio Freire klaussfre...@gmail.com wrote: On Fri, Dec 6, 2013 at 3:50 PM, Michael Bayer mike...@zzzcomputing.com wrote: +except: +self._overflow -= 1 +

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Sunil
On Friday 06 December 2013 09:45 PM, Michael Bayer wrote: [...] OK, I see this is with gevent - while I like the idea of gevent, I’m not deeply familiar with best practices for it. The QueuePool specifically uses thread-based locks to achieve it’s work. I can’t comment on what modifications

Re: [sqlalchemy] Parallel connection attempts in QueuePool

2013-12-06 Thread Sunil
On Saturday 07 December 2013 12:20 AM, Michael Bayer wrote: [...] the hard part is producing a test case. I noticed just now that even if I take overflow_lock out entirely, all the current tests pass, but this is because it’s not easy for tests to catch race conditions like that. To test the