Re: [sqlalchemy] Session / declarative_base scope

[Warwick Prince]

Hi Mike

I don’t need support very often, but when I do I know I can count on your clear 
and concise responses to save the day.

Everything was SO close - all I needed to do was take my dynamically created 
classes (SalesDocumentLine for example) and force them into the general 
globals() and hey-presto - all fixed.  ‘Knowing’ that the environment had a 
global scope, and that it was thread safe eliminated so many other possible red 
herrings.  I could then focus on why my classes could not be ‘seen’.

Thanks once again.  Best support on the interwebs. ;-)

> On 29 Sep 2016, at 11:48 PM, Mike Bayer  wrote:
> 
> 
> 
> On 09/29/2016 01:38 AM, Warwick Prince wrote:
>> Hi Mike
>> 
>> I would like a little insight into the session object, and the
>> declarative_base class.
>> 
>> I have a process running many threads, where each thread may be
>> connected to potentially a different engine/database.  If the database
>> connection between 2 or more threads is the same, then they will share
>> the same engine.  However, they each have their own MetaData objects.
>> 
>> There is a global sessionmaker() that has no binding at that time.
>> When each thread creates its OWN session, then it processes mySession =
>> Session(bind=myThreadsEngine).
>> 
>> The Engines and MetaData part has worked perfectly for years, using
>> basic queries like Table(’some_table', threadMetaData,
>> autoload=True).select().execute().fetchall(). etc.
>> 
>> I’ve started to use the ORM more now, and am using the relationships
>> between the objects.  However, I’m hitting and issue that appears to
>> centre around some shared registry or class variables or something that
>> is causing a conflict.
>> 
>> I’ve made it so each THREAD has is own Base =
>> declarative_base(metadata=theSessionsMetaData)
>> 
>> Then, classes are mapped dynamically based on this new Base, and the
>> columns are autoload’ed.  Again, this is working - sometimes.   There’s
>> some random-like problem that mostly means it does not work when I do a
>> mySession.query(myMappedClassWithRelationships) and I get the following
>> exception being raised;
> 
> so generating new classes in threads can be problematic because the registry 
> of mappers is essentially global state.   Initialization of mappers against 
> each other, which is where your error here is, is mutexed and is overall 
> thread-safe, but still, you need to make sure that all the things that your 
> class needs to be used exist.  Here, somewhere in your program you have a 
> class called SalesDocumentLine, and that class has not been seen by your 
> Python interpreter yet.   That the problem only happens randomly in threads 
> implies some kind of race condition which will make this harder to diagnose, 
> but basically that name has to exist, if your mapping refers to it.   You 
> might want to play with the configure_mappers() call that will cause this 
> initialization to occur at the point you tell it.
> 
> 
> 
> 
>> 
>>  File
>> "C:\Python27\lib\site-packages\dap-2.1.2-py2.7.egg\dap\db\dbutils.py",
>> line 323, in DAPDB_SetColumns
>>query = session.query(mappedClass).filter_by(**whereCriteria)
>>  File "build\bdist.win32\egg\sqlalchemy\orm\session.py", line 1260, in
>> query
>>return self._query_cls(entities, self, **kwargs)
>>  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 110, in
>> __init__
>>self._set_entities(entities)
>>  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 120, in
>> _set_entities
>>self._set_entity_selectables(self._entities)
>>  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 150, in
>> _set_entity_selectables
>>ent.setup_entity(*d[entity])
>>  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 3446, in
>> setup_entity
>>self._with_polymorphic = ext_info.with_polymorphic_mappers
>>  File "build\bdist.win32\egg\sqlalchemy\util\langhelpers.py", line 754,
>> in __get__
>>obj.__dict__[self.__name__] = result = self.fget(obj)
>>  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1891, in
>> _with_polymorphic_mappers
>>configure_mappers()
>>  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 2768, in
>> configure_mappers
>>mapper._post_configure_properties()
>>  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1708, in
>> _post_configure_properties
>>prop.init()
>>  File "build\bdist.win32\egg\sqlalchemy\orm\interfaces.py", line 183,
>> in init
>>self.do_init()
>>  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line
>> 1628, in do_init
>>self._process_dependent_arguments()
>>  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line
>> 1653, in _process_dependent_arguments
>>setattr(self, attr, attr_value())
>>  File
>> "build\bdist.win32\egg\sqlalchemy\ext\declarative\clsregistry.py", line
>> 293, in __call__
>>(self.prop.parent, self.arg, n.args[0], self.cls)
>> InvalidRequestError: When initializing mapper

Re: [sqlalchemy] Session / declarative_base scope

[Mike Bayer]

On 09/29/2016 01:38 AM, Warwick Prince wrote:

Hi Mike

I would like a little insight into the session object, and the
declarative_base class.

I have a process running many threads, where each thread may be
connected to potentially a different engine/database.  If the database
connection between 2 or more threads is the same, then they will share
the same engine.  However, they each have their own MetaData objects.

There is a global sessionmaker() that has no binding at that time.
When each thread creates its OWN session, then it processes mySession =
Session(bind=myThreadsEngine).

The Engines and MetaData part has worked perfectly for years, using
basic queries like Table(’some_table', threadMetaData,
autoload=True).select().execute().fetchall(). etc.

I’ve started to use the ORM more now, and am using the relationships
between the objects.  However, I’m hitting and issue that appears to
centre around some shared registry or class variables or something that
is causing a conflict.

I’ve made it so each THREAD has is own Base =
declarative_base(metadata=theSessionsMetaData)

Then, classes are mapped dynamically based on this new Base, and the
columns are autoload’ed.  Again, this is working - sometimes.   There’s
some random-like problem that mostly means it does not work when I do a
mySession.query(myMappedClassWithRelationships) and I get the following
exception being raised;


so generating new classes in threads can be problematic because the 
registry of mappers is essentially global state.   Initialization of 
mappers against each other, which is where your error here is, is 
mutexed and is overall thread-safe, but still, you need to make sure 
that all the things that your class needs to be used exist.  Here, 
somewhere in your program you have a class called SalesDocumentLine, and 
that class has not been seen by your Python interpreter yet.   That the 
problem only happens randomly in threads implies some kind of race 
condition which will make this harder to diagnose, but basically that 
name has to exist, if your mapping refers to it.   You might want to 
play with the configure_mappers() call that will cause this 
initialization to occur at the point you tell it.







  File
"C:\Python27\lib\site-packages\dap-2.1.2-py2.7.egg\dap\db\dbutils.py",
line 323, in DAPDB_SetColumns
query = session.query(mappedClass).filter_by(**whereCriteria)
  File "build\bdist.win32\egg\sqlalchemy\orm\session.py", line 1260, in
query
return self._query_cls(entities, self, **kwargs)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 110, in
__init__
self._set_entities(entities)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 120, in
_set_entities
self._set_entity_selectables(self._entities)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 150, in
_set_entity_selectables
ent.setup_entity(*d[entity])
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 3446, in
setup_entity
self._with_polymorphic = ext_info.with_polymorphic_mappers
  File "build\bdist.win32\egg\sqlalchemy\util\langhelpers.py", line 754,
in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1891, in
_with_polymorphic_mappers
configure_mappers()
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 2768, in
configure_mappers
mapper._post_configure_properties()
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1708, in
_post_configure_properties
prop.init()
  File "build\bdist.win32\egg\sqlalchemy\orm\interfaces.py", line 183,
in init
self.do_init()
  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line
1628, in do_init
self._process_dependent_arguments()
  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line
1653, in _process_dependent_arguments
setattr(self, attr, attr_value())
  File
"build\bdist.win32\egg\sqlalchemy\ext\declarative\clsregistry.py", line
293, in __call__
(self.prop.parent, self.arg, n.args[0], self.cls)
InvalidRequestError: When initializing mapper
Mapper|SalesDocument|rm_dt_documents, expression
'SalesDocumentLine.parentID==SalesDocument.id' failed to locate a name
("name 'SalesDocumentLine' is not defined"). If this is a class name,
consider adding this relationship() to the  class after both dependent classes
have been defined.

I understand what this is trying to tell me, however, the classes ARE
defined.  Sometimes the code works perfectly, but mostly not.  If I have
ONE Thread working and then start up another using exactly the same
code, then it will probably NOT work but more importantly, the one that
WAS working then dies with the same error.  Clearly something somewhere
is shared - I just can’t find out what it is, or how I can separate the
code further.

In summary;

one global sessionmaker()
global Session=sessionmaker()
each thread (for the example here) shares an Engine
each thread has it’s OWN 

[sqlalchemy] Session / declarative_base scope

[Warwick Prince]

Hi Mike

I would like a little insight into the session object, and the declarative_base 
class.

I have a process running many threads, where each thread may be connected to 
potentially a different engine/database.  If the database connection between 2 
or more threads is the same, then they will share the same engine.  However, 
they each have their own MetaData objects.   

There is a global sessionmaker() that has no binding at that time.   When each 
thread creates its OWN session, then it processes mySession = 
Session(bind=myThreadsEngine).

The Engines and MetaData part has worked perfectly for years, using basic 
queries like Table(’some_table', threadMetaData, 
autoload=True).select().execute().fetchall(). etc.

I’ve started to use the ORM more now, and am using the relationships between 
the objects.  However, I’m hitting and issue that appears to centre around some 
shared registry or class variables or something that is causing a conflict.

I’ve made it so each THREAD has is own Base = 
declarative_base(metadata=theSessionsMetaData)

Then, classes are mapped dynamically based on this new Base, and the columns 
are autoload’ed.  Again, this is working - sometimes.   There’s some 
random-like problem that mostly means it does not work when I do a 
mySession.query(myMappedClassWithRelationships) and I get the following 
exception being raised;

  File "C:\Python27\lib\site-packages\dap-2.1.2-py2.7.egg\dap\db\dbutils.py", 
line 323, in DAPDB_SetColumns
query = session.query(mappedClass).filter_by(**whereCriteria)
  File "build\bdist.win32\egg\sqlalchemy\orm\session.py", line 1260, in query
return self._query_cls(entities, self, **kwargs)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 110, in __init__
self._set_entities(entities)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 120, in 
_set_entities
self._set_entity_selectables(self._entities)
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 150, in 
_set_entity_selectables
ent.setup_entity(*d[entity])
  File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 3446, in 
setup_entity
self._with_polymorphic = ext_info.with_polymorphic_mappers
  File "build\bdist.win32\egg\sqlalchemy\util\langhelpers.py", line 754, in 
__get__
obj.__dict__[self.__name__] = result = self.fget(obj)
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1891, in 
_with_polymorphic_mappers
configure_mappers()
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 2768, in 
configure_mappers
mapper._post_configure_properties()
  File "build\bdist.win32\egg\sqlalchemy\orm\mapper.py", line 1708, in 
_post_configure_properties
prop.init()
  File "build\bdist.win32\egg\sqlalchemy\orm\interfaces.py", line 183, in init
self.do_init()
  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line 1628, in 
do_init
self._process_dependent_arguments()
  File "build\bdist.win32\egg\sqlalchemy\orm\relationships.py", line 1653, in 
_process_dependent_arguments
setattr(self, attr, attr_value())
  File "build\bdist.win32\egg\sqlalchemy\ext\declarative\clsregistry.py", line 
293, in __call__
(self.prop.parent, self.arg, n.args[0], self.cls)
InvalidRequestError: When initializing mapper 
Mapper|SalesDocument|rm_dt_documents, expression 
'SalesDocumentLine.parentID==SalesDocument.id' failed to locate a name ("name 
'SalesDocumentLine' is not defined"). If this is a class name, consider adding 
this relationship() to the  class after 
both dependent classes have been defined.

I understand what this is trying to tell me, however, the classes ARE defined.  
Sometimes the code works perfectly, but mostly not.  If I have ONE Thread 
working and then start up another using exactly the same code, then it will 
probably NOT work but more importantly, the one that WAS working then dies with 
the same error.  Clearly something somewhere is shared - I just can’t find out 
what it is, or how I can separate the code further.

In summary;

one global sessionmaker()
global Session=sessionmaker()
each thread (for the example here) shares an Engine
each thread has it’s OWN session from mySession = Session(bind=e)
each thread has it’s own Base created from 
declarative_base(metadata=threadsMetaData)
I’m declaring two classes in this example. SalesDocument and SalesDocumentLine. 
 The relationships are set up and ‘can’ work on occasion.

In that error, where exactly are they not ‘defined’.  I’ve looked in 
Base.decl_class_registry and both those names are there!  Where else do they 
need to be to be considered ‘declared'?

Any pointers as to the error of my ways would be most appreciated.  :-)

Cheers
Warwick




-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to