Re: [sqlalchemy] Is there a way to create custom classes that inherit from sqlalchemy models in external library?

[Mike Bayer]

I think it's OK to add __abstract__ to those mixins, in this case it 
seems like the (pretty complicated) class graph is adding up to 
something that causes declarative to see AbstractCar as mapped class 
thus a "car" table is added.  Adding __abstract__ to AbstractCoupe seems 
to resolve.





On 06/27/2016 05:10 PM, Angie Ellis wrote:

I implemented the changes you suggested, but I am running into errors
with some polymorphic relations. Using a different theme, here is an
example:


class AbstractVehicle(object):
__tablename__ = "vehicles"

id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
type = Column(String)

__mapper_args__ = {"polymorphic_on": type}


class AbstractCar(AbstractVehicle):
__tablename__ = "cars"

id = Column("id", Integer, primary_key=True)

__mapper_args__ = {"polymorphic_identity": "car"}
__table_args__ = (
ForeignKeyConstraint(["id"], ["vehicles.id "]),
)


class AbstractCoupe(AbstractCar):
__mapper_args__ = {"polymorphic_identity": "coupe"}


Base = declarative_base()


class Vehicle(AbstractVehicle, Base):
pass


class Car(AbstractCar, Vehicle):
pass


class Coupe(AbstractCoupe, Car):
pass


This resulted in the following error:


  File ".../model/vehicle.py", line 255, in 
class Coupe(AbstractCoupe, Car):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/api.py",
line 55, in __init__
_as_declarative(cls, classname, cls.__dict__)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
88, in _as_declarative
_MapperConfig.setup_mapping(cls, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
103, in setup_mapping
cfg_cls(cls_, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
131, in __init__
self._setup_table()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
395, in _setup_table
**table_kw)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py",
line 398, in __new__
"existing Table object." % key)
InvalidRequestError: Table 'cars' is already defined for this MetaData
instance.  Specify 'extend_existing=True' to redefine options and
columns on an existing Table object.


After adding "extend_existing=True" to the AbstractCoupe model, I
received this error:


  File ".../model/vehicle.py", line 255, in 
class Coupe(AbstractCoupe, Car):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/api.py",
line 55, in __init__
_as_declarative(cls, classname, cls.__dict__)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
88, in _as_declarative
_MapperConfig.setup_mapping(cls, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
103, in setup_mapping
cfg_cls(cls_, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
135, in __init__
self._early_mapping()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
138, in _early_mapping
self.map()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line
530, in map
**self.mapper_args
  File "", line 2, in mapper
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",
line 627, in __init__
self._configure_properties()
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",
line 1292, in _configure_properties
self._adapt_inherited_property(key, prop, False)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",
line 1514, in _adapt_inherited_property
self._configure_property(key, prop, init=False, setparent=False)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",
line 1541, in _configure_property
prop.columns[0])
  File
".../env/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py", line
476, in corresponding_column
if self.c.contains_column(column):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
line 754, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py", line
553, in columns
return self._columns.as_immutable()
AttributeError: 'Table' object has no attribute '_columns'


Is there a way to fix or work around this?

Thanks,
Angie

On Wed, Jun 22, 2016 at 7:27 PM, Mike Bayer > wrote:



On 06/22/2016 04:53 PM, Angie Ellis wrote:

Mike,

Thank you for the response! I do have control over the external
library.
I like your idea having both the abstract classes and concrete
classes
available in the library because I have use cases for both in my
  

Re: [sqlalchemy] Is there a way to create custom classes that inherit from sqlalchemy models in external library?

[Angie Ellis]

I implemented the changes you suggested, but I am running into errors with
some polymorphic relations. Using a different theme, here is an example:


class AbstractVehicle(object):
__tablename__ = "vehicles"

id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
type = Column(String)

__mapper_args__ = {"polymorphic_on": type}


class AbstractCar(AbstractVehicle):
__tablename__ = "cars"

id = Column("id", Integer, primary_key=True)

__mapper_args__ = {"polymorphic_identity": "car"}
__table_args__ = (
ForeignKeyConstraint(["id"], ["vehicles.id"]),
)


class AbstractCoupe(AbstractCar):
__mapper_args__ = {"polymorphic_identity": "coupe"}


Base = declarative_base()


class Vehicle(AbstractVehicle, Base):
pass


class Car(AbstractCar, Vehicle):
pass


class Coupe(AbstractCoupe, Car):
pass


This resulted in the following error:


  File ".../model/vehicle.py", line 255, in 
class Coupe(AbstractCoupe, Car):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/api.py",
line 55, in __init__
_as_declarative(cls, classname, cls.__dict__)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 88, in _as_declarative
_MapperConfig.setup_mapping(cls, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 103, in setup_mapping
cfg_cls(cls_, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 131, in __init__
self._setup_table()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 395, in _setup_table
**table_kw)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line
398, in __new__
"existing Table object." % key)
InvalidRequestError: Table 'cars' is already defined for this MetaData
instance.  Specify 'extend_existing=True' to redefine options and columns
on an existing Table object.


After adding "extend_existing=True" to the AbstractCoupe model, I received
this error:


  File ".../model/vehicle.py", line 255, in 
class Coupe(AbstractCoupe, Car):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/api.py",
line 55, in __init__
_as_declarative(cls, classname, cls.__dict__)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 88, in _as_declarative
_MapperConfig.setup_mapping(cls, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 103, in setup_mapping
cfg_cls(cls_, classname, dict_)
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 135, in __init__
self._early_mapping()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 138, in _early_mapping
self.map()
  File
".../env/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py",
line 530, in map
**self.mapper_args
  File "", line 2, in mapper
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line
627, in __init__
self._configure_properties()
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line
1292, in _configure_properties
self._adapt_inherited_property(key, prop, False)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line
1514, in _adapt_inherited_property
self._configure_property(key, prop, init=False, setparent=False)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line
1541, in _configure_property
prop.columns[0])
  File ".../env/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py",
line 476, in corresponding_column
if self.c.contains_column(column):
  File
".../env/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line
754, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
  File ".../env/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py",
line 553, in columns
return self._columns.as_immutable()
AttributeError: 'Table' object has no attribute '_columns'


Is there a way to fix or work around this?

Thanks,
Angie

On Wed, Jun 22, 2016 at 7:27 PM, Mike Bayer 
wrote:

>
>
> On 06/22/2016 04:53 PM, Angie Ellis wrote:
>
>> Mike,
>>
>> Thank you for the response! I do have control over the external library.
>> I like your idea having both the abstract classes and concrete classes
>> available in the library because I have use cases for both in my
>> applications. However, I'm not quite sure how to configure the
>> inheritance and declarative base. What am I doing wrong here?
>>
>>
>> class AbstractFoo(object):
>> __abstract__ = True
>> __tablename__ = "foo"
>> id = Column(Integer, primary_key=True)
>> type = Column(String)
>> foo_bar_id = Column(Integer, ForeignKey("foo_bar.id
>> "))
>> foo_bar = relationship("FooBar", 

Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

On 06/27/2016 12:46 PM, davepuffin via sqlalchemy wrote:

I've checked netstat and there are only 15 mysql connections (status is
still reporting the same):
Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35

On our other servers, the number of connections (from netstat) matches
up with the pool status as expected.
I will try and recreate the problem with a test.

Many thanks!



there's almost certainly a bug on our end on that, then.






On Monday, June 27, 2016 at 4:36:21 PM UTC+1, Mike Bayer wrote:

Well it is possible QueuePool doesn't do the right thing under
eventlet,
though we use it a lot w/ eventlet in Openstack and I've tested it
myself quite a bit.  Have not peeked at this number, however.

You'd need to produce for me a self contained test suite that
illustrates this behavior occurring, that would be very helpful in
order
to track this down.

Do you see too many actual database connections in play when this
condition occurs ?  That is, using netstat you'd see more than 120
connections.



On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote:
> Thanks for the reply :)
>
> I think you're right in that it must be threading related somehow,
but
> it looks like all the monkeypatching has been applied correctly. If I
> examine the locking object used in the pool I get:
>
 type(engine.pool._overflow_lock)
>
> 
>
>
> On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer
 > wrote:
>
>
>
> On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:
>
> Hi
>
> QueuePool.status() is returning some odd looking negative
values
> from
> one of our production applications.
>
> engine.pool.status()
>
> 'Pool size: 20 Connections in pool: 19 Current Overflow:
-36 Current
> Checked out connections: -35'
>
> The pool was initialised with:  pool_size=20,
max_overflow=100.
>
>
> Does anyone know of any scenarios that could cause this?
>
>
> We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
> The application runs in a single python process that uses
> eventlet to
> manage many greenthreads.
>
>
>
> well that's very likely to be related, are you monkeypatching all
> thread / Lock primitives before importing SQLAlchemy?  QueuePool
> relies upon traditional thread and locking primitives which will
> fail if you use them unpatched in a green-thread situation.   The
> Queue here is basically Python's Queue.Queue so it appears
like the
> queue has been populated with more objects than it is supposed
to allow.
>
>
>
> The situation appears to occur sometime after the overflow
has
> been used.
>
> Thanks
>
> --
> You received this message because you are subscribed to
the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to sqlalchemy+...@googlegroups.com 
> 
>  >.
> To post to this group, send email to
sqlal...@googlegroups.com 
> 
>  >.
> Visit this group at
https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout
.
>
>
> --
> You received this message because you are subscribed to a
topic in
> the Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
>
https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe
.
> To unsubscribe from this group and all its topics, send an
email to
> sqlalchemy+...@googlegroups.com 
> .
> To post to this group, send email to sqlal...@googlegroups.com

> .
> Visit this group at https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout

Re: [sqlalchemy] QueuePool negative overflow

[davepuffin via sqlalchemy]

I've checked netstat and there are only 15 mysql connections (status is 
still reporting the same): 
Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35

On our other servers, the number of connections (from netstat) matches up 
with the pool status as expected.
I will try and recreate the problem with a test.

Many thanks!



On Monday, June 27, 2016 at 4:36:21 PM UTC+1, Mike Bayer wrote:
>
> Well it is possible QueuePool doesn't do the right thing under eventlet, 
> though we use it a lot w/ eventlet in Openstack and I've tested it 
> myself quite a bit.  Have not peeked at this number, however. 
>
> You'd need to produce for me a self contained test suite that 
> illustrates this behavior occurring, that would be very helpful in order 
> to track this down. 
>
> Do you see too many actual database connections in play when this 
> condition occurs ?  That is, using netstat you'd see more than 120 
> connections. 
>
>
>
> On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote: 
> > Thanks for the reply :) 
> > 
> > I think you're right in that it must be threading related somehow, but 
> > it looks like all the monkeypatching has been applied correctly. If I 
> > examine the locking object used in the pool I get: 
> > 
>  type(engine.pool._overflow_lock) 
> > 
> >  
> > 
> > 
> > On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer   
> > > wrote: 
> > 
> > 
> > 
> > On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote: 
> > 
> > Hi 
> > 
> > QueuePool.status() is returning some odd looking negative values 
> > from 
> > one of our production applications. 
> > 
> > engine.pool.status() 
> > 
> > 'Pool size: 20 Connections in pool: 19 Current Overflow: -36 
> Current 
> > Checked out connections: -35' 
> > 
> > The pool was initialised with:  pool_size=20, max_overflow=100. 
> > 
> > 
> > Does anyone know of any scenarios that could cause this? 
> > 
> > 
> > We are using SQLAlchemy==1.0.13 connecting to MySql 5.6. 
> > The application runs in a single python process that uses 
> > eventlet to 
> > manage many greenthreads. 
> > 
> > 
> > 
> > well that's very likely to be related, are you monkeypatching all 
> > thread / Lock primitives before importing SQLAlchemy?  QueuePool 
> > relies upon traditional thread and locking primitives which will 
> > fail if you use them unpatched in a green-thread situation.   The 
> > Queue here is basically Python's Queue.Queue so it appears like the 
> > queue has been populated with more objects than it is supposed to 
> allow. 
> > 
> > 
> > 
> > The situation appears to occur sometime after the overflow has 
> > been used. 
> > 
> > Thanks 
> > 
> > -- 
> > You received this message because you are subscribed to the 
> Google 
> > Groups "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from 
> > it, send 
> > an email to sqlalchemy+...@googlegroups.com  
> >  
>
> >  > >. 
>
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> >  
> >  > >. 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to a topic in 
> > the Google Groups "sqlalchemy" group. 
> > To unsubscribe from this topic, visit 
> > https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe. 
>
> > To unsubscribe from this group and all its topics, send an email to 
> > sqlalchemy+...@googlegroups.com  
> > . 
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to sqlalchemy+...@googlegroups.com  
> > . 
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more 

Re: [sqlalchemy] Re: Tracking query performance per Session

[Martijn van Oosterhout]

Ah, thanks for the tip, the info fields work nicely as well.

Have a nice day,

On Monday, June 27, 2016 at 4:09:54 PM UTC+2, Mike Bayer wrote:
>
>
> Hi Martijn - 
>
> Sorry I missed this post.Your solution is pretty good.  For the 
> setattr() stuff, both Connection and Session have an .info attribute 
> which is a dictionary you can use for memoizations. 
>
>
-- 
Martijn van Oosterhout  http://svana.org/kleptog/

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

Well it is possible QueuePool doesn't do the right thing under eventlet, 
though we use it a lot w/ eventlet in Openstack and I've tested it 
myself quite a bit.  Have not peeked at this number, however.


You'd need to produce for me a self contained test suite that 
illustrates this behavior occurring, that would be very helpful in order 
to track this down.


Do you see too many actual database connections in play when this 
condition occurs ?  That is, using netstat you'd see more than 120 
connections.




On 06/27/2016 11:30 AM, 'Dave Puffin' via sqlalchemy wrote:

Thanks for the reply :)

I think you're right in that it must be threading related somehow, but
it looks like all the monkeypatching has been applied correctly. If I
examine the locking object used in the pool I get:


type(engine.pool._overflow_lock)





On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer > wrote:



On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:

Hi

QueuePool.status() is returning some odd looking negative values
from
one of our production applications.

engine.pool.status()

'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses
eventlet to
manage many greenthreads.



well that's very likely to be related, are you monkeypatching all
thread / Lock primitives before importing SQLAlchemy?  QueuePool
relies upon traditional thread and locking primitives which will
fail if you use them unpatched in a green-thread situation.   The
Queue here is basically Python's Queue.Queue so it appears like the
queue has been populated with more objects than it is supposed to allow.



The situation appears to occur sometime after the overflow has
been used.

Thanks

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from
it, send
an email to sqlalchemy+unsubscr...@googlegroups.com

>.
To post to this group, send email to sqlalchemy@googlegroups.com

>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to a topic in
the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] QueuePool negative overflow

['Dave Puffin' via sqlalchemy]

Thanks for the reply :)

I think you're right in that it must be threading related somehow, but it
looks like all the monkeypatching has been applied correctly. If I examine
the locking object used in the pool I get:

>>> type(engine.pool._overflow_lock)



On Mon, Jun 27, 2016 at 4:21 PM, Mike Bayer 
wrote:

>
>
> On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:
>
>> Hi
>>
>> QueuePool.status() is returning some odd looking negative values from
>> one of our production applications.
>>
>> engine.pool.status()
>
 'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
>> Checked out connections: -35'
>>
>> The pool was initialised with:  pool_size=20, max_overflow=100.
>>
>>
>> Does anyone know of any scenarios that could cause this?
>>
>>
>> We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
>> The application runs in a single python process that uses eventlet to
>> manage many greenthreads.
>>
>
>
> well that's very likely to be related, are you monkeypatching all thread /
> Lock primitives before importing SQLAlchemy?  QueuePool relies upon
> traditional thread and locking primitives which will fail if you use them
> unpatched in a green-thread situation.   The Queue here is basically
> Python's Queue.Queue so it appears like the queue has been populated with
> more objects than it is supposed to allow.
>
>
>
>> The situation appears to occur sometime after the overflow has been used.
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to sqlalchemy+unsubscr...@googlegroups.com
>> .
>> To post to this group, send email to sqlalchemy@googlegroups.com
>> .
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/UWxi-bknpZE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] QueuePool negative overflow

[Mike Bayer]

On 06/27/2016 10:48 AM, davepuffin via sqlalchemy wrote:

Hi

QueuePool.status() is returning some odd looking negative values from
one of our production applications.


engine.pool.status()

'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses eventlet to
manage many greenthreads.



well that's very likely to be related, are you monkeypatching all thread 
/ Lock primitives before importing SQLAlchemy?  QueuePool relies upon 
traditional thread and locking primitives which will fail if you use 
them unpatched in a green-thread situation.   The Queue here is 
basically Python's Queue.Queue so it appears like the queue has been 
populated with more objects than it is supposed to allow.





The situation appears to occur sometime after the overflow has been used.

Thanks

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] QueuePool negative overflow

[davepuffin via sqlalchemy]

Hi

QueuePool.status() is returning some odd looking negative values from one 
of our production applications.

>>> engine.pool.status() 
'Pool size: 20 Connections in pool: 19 Current Overflow: -36 Current 
Checked out connections: -35'

The pool was initialised with:  pool_size=20, max_overflow=100.


Does anyone know of any scenarios that could cause this?


We are using SQLAlchemy==1.0.13 connecting to MySql 5.6.
The application runs in a single python process that uses eventlet to 
manage many greenthreads.

The situation appears to occur sometime after the overflow has been used.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] 1.1.0b1: construct has no default compilation handler.

[Mike Bayer]

Not really sure, that URL itself won't allow a psycopg2 connection to 
even occur.   The error means that the construct is trying to be 
generated as a string, like in a print statement.   But 1.1 has a new 
feature that allows default stringification of pg.ARRAY and other 
constructs to actually work.


Need a stack trace at the very least.



On 06/27/2016 08:34 AM, Martijn van Oosterhout wrote:

from sqlalchemy import create_engine, MetaData
from sqlalchemy import Column
from sqlalchemy.dialects.postgresql import ARRAY
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base

metadata = MetaData()
metadata.bind = create_engine('postgresql:///')
Base = declarative_base(metadata=metadata)

class Test(Base):
__tablename__ = 'test'
reference = Column(ARRAY(String), nullable=False,
server_default='{}', primary_key=True)

metadata.create_all(bind=metadata.bind)


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] Firebird is no way to implement UPDATE OR INSERT

[Mike Bayer]

not currently, and we don't actually spend much time working on the 
Firebird dialect (would love if someone could come take it over).   Your 
best bet would be working with custom compilation rules described in 
http://docs.sqlalchemy.org/en/latest/core/compiler.html .




On 06/27/2016 07:47 AM, uralbash wrote:


Is there a way to construct a query "UPDATE OR INSERT" for firebird
like Postgres "INSERT...ON CONFLICT (Upsert)"

(http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#insert-on-conflict-upsert)?


Firebird docs
http://www.firebirdsql.org/refdocs/langrefupd25-update-or-insert.html

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] Re: Tracking query performance per Session

[Mike Bayer]

Hi Martijn -

Sorry I missed this post.Your solution is pretty good.  For the 
setattr() stuff, both Connection and Session have an .info attribute 
which is a dictionary you can use for memoizations.




On 06/27/2016 03:02 AM, Martijn van Oosterhout wrote:


On Tuesday, June 21, 2016 at 11:25:36 PM UTC+2, Martijn van Oosterhout
wrote:

Hi,

In an effort to find the bottlenecks in our system we'd like to
collect statistics about which queries take a long time, or return a
lot of rows, or paths which are triggering lots of queries.

Now, the system consists of lots of parts working in parallel, each
with their own Session object, so we want the stats per Session.


In answer to my own question, this is what I came up with:
https://gist.github.com/kleptog/1a866b67faa2c917758496a4fbbf92b8

It allows you to do things like:

s1 = SessionMaker()
s1.execute("SELECT 1")
s1.commit()
print s1._query_stats
print s1._query_stats.query_log

Output:


[(1467010565.738685, 'SELECT 1', {}, [('__main__', '', 127),
('__main__', 'test', 115)], 0.0002570152282714844)]

Using the log you have access to all the executed queries, which you can
the aggregate, save to disk, etc. It tracks start time, parameters, call
stack and duration. The adding of extra attributes to the Session and
Connection objects is not pretty, but it works.

Have a nice day,
--
Martijn van Oosterhout >
http://svana.org/kleptog/

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] 1.1.0b1: construct has no default compilation handler.

[Martijn van Oosterhout]

When trying out 1.1.0b1 (to look at the new events) I got the $SUBJECT 
error from our test cases.

I'm not sure what exactly is going on, because when I run the script below 
line-by-line in the django shell, it breaks, but if I run it from the 
command-line it works. But that's probably more to do with my local setup. 
With 1.0.0 it doesn't fail in either case.


from sqlalchemy import create_engine, MetaData
from sqlalchemy import Column
from sqlalchemy.dialects.postgresql import ARRAY
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base

metadata = MetaData()
metadata.bind = create_engine('postgresql:///')
Base = declarative_base(metadata=metadata)

class Test(Base):
__tablename__ = 'test'
reference = Column(ARRAY(String), nullable=False, server_default='{}', 
primary_key=True)

metadata.create_all(bind=metadata.bind)

===

Hope this helps,
-- 
Martijn van Oosterhout

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Firebird is no way to implement UPDATE OR INSERT

[uralbash]

Is there a way to construct a query "UPDATE OR INSERT" for firebird like 
Postgres "INSERT...ON CONFLICT (Upsert)" 
(http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#insert-on-conflict-upsert)?


Firebird docs 
http://www.firebirdsql.org/refdocs/langrefupd25-update-or-insert.html

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

[Martijn van Oosterhout]

On Thursday, June 23, 2016 at 11:22:32 PM UTC+2, Chung wrote:
>
> Ah thanks Mike; that makes sense.
>
> What we have is actually hundreds of different databases with roughly the 
> same schema, with new databases getting created all the time live.  The 
> table schemas for each database are also being altered live.  So I'm not 
> sure we can get away with not reflecting the table schema per request; any 
> request may be referencing any new database or any newly created column. 
>  Possibly there's still something meaningful we can cache?  We will think 
> about this.
>
>
That actually sounds like a situation we had. I think that if you attach 
the connection/bind to the Session instead of to the Metadata, you can use 
the same Metadata for several different databases, as long as they have the 
same schema. Unfortunately, if you didn't set it up that way in the 
beginning it can be painful to rework...

-- 
Martijn van Oosterhout

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Re: Tracking query performance per Session

[Martijn van Oosterhout]

On Tuesday, June 21, 2016 at 11:25:36 PM UTC+2, Martijn van Oosterhout 
wrote:
>
> Hi, 
>
> In an effort to find the bottlenecks in our system we'd like to collect 
> statistics about which queries take a long time, or return a lot of rows, 
> or paths which are triggering lots of queries.
>
> Now, the system consists of lots of parts working in parallel, each with 
> their own Session object, so we want the stats per Session.
>
>
In answer to my own question, this is what I came up with: 
https://gist.github.com/kleptog/1a866b67faa2c917758496a4fbbf92b8

It allows you to do things like:

s1 = SessionMaker()
s1.execute("SELECT 1")
s1.commit()
print s1._query_stats
print s1._query_stats.query_log

Output:


[(1467010565.738685, 'SELECT 1', {}, [('__main__', '', 127), 
('__main__', 'test', 115)], 0.0002570152282714844)]

Using the log you have access to all the executed queries, which you can 
the aggregate, save to disk, etc. It tracks start time, parameters, call 
stack and duration. The adding of extra attributes to the Session and 
Connection objects is not pretty, but it works.

Have a nice day,
-- 
Martijn van Oosterhout  http://svana.org/kleptog/

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.