Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

On 06/27/2016 12:46 PM, davepuffin via sqlalchemy wrote:

I've checked netstat and there are only 15 mysql connections (status is
still reporting the same):
Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35

On our other servers, the number of connections (from netstat) matches
up with the pool status as expected.
I will try and recreate the problem with a test.

Many thanks!



there's almost certainly a bug on our end on that, then.






On Monday, June 27, 2016 at 4:36:21 PM UTC+1, Mike Bayer wrote:

Well it is possible QueuePool doesn't do the right thing under
eventlet,
though we use it a lot w/ eventlet in Openstack and I've tested it
myself quite a bit.  Have not peeked at this number, however.

You'd need to produce for me a self contained test suite that
illustrates this behavior occurring, that would be very helpful in
order
to track this down.

Do you see too many actual database connections in play when this
condition occurs ?  That is, using netstat you'd see more than 120
connections.



On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote:
> Thanks for the reply :)
>
> I think you're right in that it must be threading related somehow,
but
> it looks like all the monkeypatching has been applied correctly. If I
> examine the locking object used in the pool I get:
>
 type(engine.pool._overflow_lock)
>
> 
>
>
> On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer
 > wrote:
>
>
>
> On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:
>
> Hi
>
> QueuePool.status() is returning some odd looking negative
values
> from
> one of our production applications.
>
> engine.pool.status()
>
> 'Pool size: 20 Connections in pool: 19 Current Overflow:
-36 Current
> Checked out connections: -35'
>
> The pool was initialised with:  pool_size=20,
max_overflow=100.
>
>
> Does anyone know of any scenarios that could cause this?
>
>
> We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
> The application runs in a single python process that uses
> eventlet to
> manage many greenthreads.
>
>
>
> well that's very likely to be related, are you monkeypatching all
> thread / Lock primitives before importing SQLAlchemy?  QueuePool
> relies upon traditional thread and locking primitives which will
> fail if you use them unpatched in a green-thread situation.   The
> Queue here is basically Python's Queue.Queue so it appears
like the
> queue has been populated with more objects than it is supposed
to allow.
>
>
>
> The situation appears to occur sometime after the overflow
has
> been used.
>
> Thanks
>
> --
> You received this message because you are subscribed to
the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to sqlalchemy+...@googlegroups.com 
> 
>  >.
> To post to this group, send email to
sqlal...@googlegroups.com 
> 
>  >.
> Visit this group at
https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout
.
>
>
> --
> You received this message because you are subscribed to a
topic in
> the Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
>
https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe
.
> To unsubscribe from this group and all its topics, send an
email to
> sqlalchemy+...@googlegroups.com 
> .
> To post to this group, send email to sqlal...@googlegroups.com

> .
> Visit this group at https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout

Re: [sqlalchemy] QueuePool negative overflow

[davepuffin via sqlalchemy]

I've checked netstat and there are only 15 mysql connections (status is 
still reporting the same): 
Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35

On our other servers, the number of connections (from netstat) matches up 
with the pool status as expected.
I will try and recreate the problem with a test.

Many thanks!



On Monday, June 27, 2016 at 4:36:21 PM UTC+1, Mike Bayer wrote:
>
> Well it is possible QueuePool doesn't do the right thing under eventlet, 
> though we use it a lot w/ eventlet in Openstack and I've tested it 
> myself quite a bit.  Have not peeked at this number, however. 
>
> You'd need to produce for me a self contained test suite that 
> illustrates this behavior occurring, that would be very helpful in order 
> to track this down. 
>
> Do you see too many actual database connections in play when this 
> condition occurs ?  That is, using netstat you'd see more than 120 
> connections. 
>
>
>
> On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote: 
> > Thanks for the reply :) 
> > 
> > I think you're right in that it must be threading related somehow, but 
> > it looks like all the monkeypatching has been applied correctly. If I 
> > examine the locking object used in the pool I get: 
> > 
>  type(engine.pool._overflow_lock) 
> > 
> >  
> > 
> > 
> > On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer   
> > > wrote: 
> > 
> > 
> > 
> > On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote: 
> > 
> > Hi 
> > 
> > QueuePool.status() is returning some odd looking negative values 
> > from 
> > one of our production applications. 
> > 
> > engine.pool.status() 
> > 
> > 'Pool size: 20 Connections in pool: 19 Current Overflow: -36 
> Current 
> > Checked out connections: -35' 
> > 
> > The pool was initialised with:  pool_size=20, max_overflow=100. 
> > 
> > 
> > Does anyone know of any scenarios that could cause this? 
> > 
> > 
> > We are using SQLAlchemy==1.0.13 connecting to MySql 5.6. 
> > The application runs in a single python process that uses 
> > eventlet to 
> > manage many greenthreads. 
> > 
> > 
> > 
> > well that's very likely to be related, are you monkeypatching all 
> > thread / Lock primitives before importing SQLAlchemy?  QueuePool 
> > relies upon traditional thread and locking primitives which will 
> > fail if you use them unpatched in a green-thread situation.   The 
> > Queue here is basically Python's Queue.Queue so it appears like the 
> > queue has been populated with more objects than it is supposed to 
> allow. 
> > 
> > 
> > 
> > The situation appears to occur sometime after the overflow has 
> > been used. 
> > 
> > Thanks 
> > 
> > -- 
> > You received this message because you are subscribed to the 
> Google 
> > Groups "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from 
> > it, send 
> > an email to sqlalchemy+...@googlegroups.com  
> >  
>
> >  > >. 
>
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> >  
> >  > >. 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to a topic in 
> > the Google Groups "sqlalchemy" group. 
> > To unsubscribe from this topic, visit 
> > https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe. 
>
> > To unsubscribe from this group and all its topics, send an email to 
> > sqlalchemy+...@googlegroups.com  
> > . 
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to sqlalchemy+...@googlegroups.com  
> > . 
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more 

Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

Well it is possible QueuePool doesn't do the right thing under eventlet, 
though we use it a lot w/ eventlet in Openstack and I've tested it 
myself quite a bit.  Have not peeked at this number, however.


You'd need to produce for me a self contained test suite that 
illustrates this behavior occurring, that would be very helpful in order 
to track this down.


Do you see too many actual database connections in play when this 
condition occurs ?  That is, using netstat you'd see more than 120 
connections.




On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote:

Thanks for the reply :)

I think you're right in that it must be threading related somehow, but
it looks like all the monkeypatching has been applied correctly. If I
examine the locking object used in the pool I get:


type(engine.pool._overflow_lock)





On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer > wrote:



On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:

Hi

QueuePool.status() is returning some odd looking negative values
from
one of our production applications.

engine.pool.status()

'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses
eventlet to
manage many greenthreads.



well that's very likely to be related, are you monkeypatching all
thread / Lock primitives before importing SQLAlchemy?  QueuePool
relies upon traditional thread and locking primitives which will
fail if you use them unpatched in a green-thread situation.   The
Queue here is basically Python's Queue.Queue so it appears like the
queue has been populated with more objects than it is supposed to allow.



The situation appears to occur sometime after the overflow has
been used.

Thanks

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from
it, send
an email to sqlalchemy+unsubscr...@googlegroups.com

>.
To post to this group, send email to sqlalchemy@googlegroups.com

>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to a topic in
the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] QueuePool negative overflow

['Dave Puffin' via sqlalchemy]

Thanks for the reply :)

I think you're right in that it must be threading related somehow, but it
looks like all the monkeypatching has been applied correctly. If I examine
the locking object used in the pool I get:

>>> type(engine.pool._overflow_lock)



On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer 
wrote:

>
>
> On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:
>
>> Hi
>>
>> QueuePool.status() is returning some odd looking negative values from
>> one of our production applications.
>>
>> engine.pool.status()
>
 'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
>> Checked out connections: -35'
>>
>> The pool was initialised with:  pool_size=20, max_overflow=100.
>>
>>
>> Does anyone know of any scenarios that could cause this?
>>
>>
>> We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
>> The application runs in a single python process that uses eventlet to
>> manage many greenthreads.
>>
>
>
> well that's very likely to be related, are you monkeypatching all thread /
> Lock primitives before importing SQLAlchemy?  QueuePool relies upon
> traditional thread and locking primitives which will fail if you use them
> unpatched in a green-thread situation.   The Queue here is basically
> Python's Queue.Queue so it appears like the queue has been populated with
> more objects than it is supposed to allow.
>
>
>
>> The situation appears to occur sometime after the overflow has been used.
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to sqlalchemy+unsubscr...@googlegroups.com
>> .
>> To post to this group, send email to sqlalchemy@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:

Hi

QueuePool.status() is returning some odd looking negative values from
one of our production applications.


engine.pool.status()

'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses eventlet to
manage many greenthreads.



well that's very likely to be related, are you monkeypatching all thread 
/ Lock primitives before importing SQLAlchemy?  QueuePool relies upon 
traditional thread and locking primitives which will fail if you use 
them unpatched in a green-thread situation.   The Queue here is 
basically Python's Queue.Queue so it appears like the queue has been 
populated with more objects than it is supposed to allow.





The situation appears to occur sometime after the overflow has been used.

Thanks

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] QueuePool negative overflow

[davepuffin via sqlalchemy]

Hi

QueuePool.status() is returning some odd looking negative values from one 
of our production applications.

>>> engine.pool.status() 
'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current 
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses eventlet to 
manage many greenthreads.

The situation appears to occur sometime after the overflow has been used.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.