I forgot to mention, we’re not using clear_mappers().
Thanks,
Will
> On 21 Feb 2016, at 19:19, Will Angenent <w.angen...@gmail.com> wrote:
>
> Hi Mike,
>
> Thanks for your quick response yet again! Here’s the stack trace.
>
> tests/integration/test_database.py:14: in test_database_is_up_to_date
> create_test_db(session)
> __init__.py:111: in create_test_db
> pd_utils.do_import(dtype='locations', ifile=yaml_file)
> ../utils/provider_data/__init__.py:54: in do_import
> inserted, updated = getattr(self, 'import_%s' % item)(ifile)
> ../utils/provider_data/__init__.py:22: in import_locations
> return import_locations(self.session, ifile)
> ../utils/provider_data/locations.py:190: in import_locations
> Location).filter_by(
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py:1260:
> in query
> return self._query_cls(entities, self, **kwargs)
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:110:
> in __init__
> self._set_entities(entities)
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:120:
> in _set_entities
> self._set_entity_selectables(self._entities)
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:150:
> in _set_entity_selectables
> ent.setup_entity(*d[entity])
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py:3250:
> in setup_entity
> self._with_polymorphic = ext_info.with_polymorphic_mappers
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py:747:
> in __get__
> obj.__dict__[self.__name__] = result = self.fget(obj)
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:1893:
> in _with_polymorphic_mappers
> configure_mappers()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:2756:
> in configure_mappers
> mapper._post_configure_properties()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py:1710:
> in _post_configure_properties
> prop.init()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/interfaces.py:183:
> in init
> self.do_init()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1613:
> in do_init
> self._setup_join_conditions()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1688:
> in _setup_join_conditions
> can_be_synced_fn=self._columns_are_mapped
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:1956:
> in __init__
> self._determine_joins()
> ../../devenv/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py:2060:
> in _determine_joins
> "specify a 'primaryjoin' expression." % self.prop)
> E NoForeignKeysError: Could not determine join condition between
> parent/child tables on relationship Resource.tags - there are no foreign keys
> linking these tables. Ensure that referencing columns are associated with a
> ForeignKey or ForeignKeyConstraint, or specify a 'primaryjoin' expression.
>
>> The sys.modules activity is not really the primary cause, it's that alembic
>> makes use of a module object in a temporary way.
> Absolutely agree. What I did to diagnose this was to replace
> secondary=lambda: tag_to_resource with secondary=non_lambda_tag_to_resource,
> using this:
>
> def non_lambda_tag_to_resource():
> import sys
> sys.stderr.write('*** tag_to_resource=%s\n' % tag_to_resource)
> #
> sys.stderr.write('*** name=%s\n' % __name__)
> #
> return tag_to_resource
>
> What I found is that in the bad case, both tag_to_resource and __name__ were
> None.
>
> Thanks,
> Will
>
>> On 21 Feb 2016, at 19:12, Mike Bayer <clas...@zzzcomputing.com
>> <mailto:clas...@zzzcomputing.com>> wrote:
>>
>>
>>
>> Hi there -
>>
>> Can you post a stack trace, and also is your test suite making use of
>> clear_mappers() ?
>>
>> The sys.modules activity is not really the primary cause, it's that alembic
>> makes use of a module object in a temporary way.
>>
>> On Feb 21, 2016, at 1:48 PM, Will Angenent <w.angen...@gmail.com
>> <mailto:w.angen...@gmail.com>> wrote:
>>
>>> Hi,
>>>
>>> We had this interesting issue recently, and I've been trying to figure out
>>> if we deserve this, if this is simply unavoidable, or whether it can be
>>> considered a bug. We're using python 2.7.6, sqlalchemy 1.0.12 and alembic
>>> 0.8.4.
>>>
>>> Summary:
>>>
>>> This statement in alembic.util.pyfiles.load_python_file():
>>> del sys.modules[module_id]
>>> randomly causes the reference count of the module object to become zero;
>>> triggering cleanup of the object. This effectively causes all variables in
>>> the migration file to become None, leading to an sqlalchemy mapper problem
>>> initializing a mapper configuration for a many-to-many relationship in a
>>> model defined in the migration file.
>>>
>>> Are we being stupid to be using the ORM in alembic migrations? If not, is
>>> it worth for me to spend more time on this? Is there any way to get this to
>>> behave non-randomly? More details are below.
>>>
>>> Thanks,
>>> Will
>>>
>>> Long version...
>>>
>>> What happened is that someone in my team added an alembic migration. He
>>> used the sqlalchemy ORM and used a declarative_base with a couple of model
>>> files to get the job done. The migration was fine and everyone was happy.
>>> Then, about a week later, I added an import statement in a totally
>>> unrelated area of code, and suddenly running alembic upgrade starting
>>> failing with a ORM mapper error. I didn't spend much time on it, but
>>> refactored a couple of things and the problem vanished.
>>>
>>> Then a couple of days later, our tests started failing with the same error.
>>> We had a closer look and found the failure to be random. The inclusion of
>>> the import statment seemed to trigger the random behavior. It wasn't just
>>> the import statement though, other changes, such as removing a property in
>>> an ORM class could make the problem appear or go away. What we were doing
>>> in this particualr failure mode, is running py.test which would, in order:
>>>
>>> - import this random 3rd party module
>>> - use the alembic API to upgrade to ensure a postgres database is up to date
>>> - later on, in an unrelated test, do a query, triggering the initialization
>>> of the mappings and crashing
>>>
>>> At first, I thought it might be a problem with sqlalchemy. Spurred on by
>>> this comment in mapper.py:
>>>
>>> # initialize properties on all mappers
>>> # note that _mapper_registry is unordered, which
>>> # may randomly conceal/reveal issues related to
>>> # the order of mapper compilation
>>>
>>> I added a couple of sorted() statements throughout the code, but it made no
>>> difference. Finally, I found that the problem was a lambda function in a
>>> relationship with a secondary. Something like e.g.
>>>
>>> tag_to_resource = Table(
>>> 'tag_to_resource', Base.metadata,
>>> Column('tag_id', ForeignKey('tags.id', ondelete='CASCADE'),
>>> primary_key=True, index=True),
>>> Column('resource_id', ForeignKey('resources.id', ondelete='CASCADE'),
>>> primary_key=True, index=True)
>>> )
>>>
>>> class Resource(Base):
>>> __tablename__ = 'resources'
>>> id = Column(UUIDType(binary=True), primary_key=True, default=uuid.uuid4)
>>>
>>> tags = relationship("Tag", secondary=lambda: tag_to_resource,
>>> backref='resources')
>>>
>>> The lambda function called in _process_dependent_arguments() was returning
>>> None instead of tag_to_resource. Resulting in a:
>>>
>>> sqlalchemy.exc.NoForeignKeysError: Could not determine join condition
>>> between parent/child tables on relationship Resource.tags - there are no
>>> foreign keys linking these tables. Ensure that referencing columns are
>>> associated with a ForeignKey or ForeignKeyConstraint, or specify a
>>> 'primaryjoin' expression.
>>>
>>> Looking deeper I found that __name__ was also None. This kind of thing
>>> happens when sys.modules is messed with. I looked at the alembic code and
>>> found this in load_python_file():
>>>
>>> del sys.modules[module_id]
>>>
>>> If I remove that statement, the problem goes away.
>>>
>>> Could it be that the reference count of the module object is becoming zero
>>> randomly, causing python to delete the data, as explained in this post?
>>> http://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name
>>>
>>> <http://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name>
>>>
>>> I've narrowed the problem down to a python test script, but it still
>>> imports a load of other stuff. I can trigger the good + bad case by just
>>> removing an import statement. I've been trying to get this down to a simple
>>> script in an attempt to prove what's going on, but the problem tends to
>>> come and go while I'm deleting code; making it difficult to narrow down.
>>> For example, I was convinced one day that the problem vanished by upgrading
>>> to sql alchemy 1.0.12, but the very next day the same code started failing
>>> again!
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to sqlalchemy+unsubscr...@googlegroups.com
>>> <mailto:sqlalchemy+unsubscr...@googlegroups.com>.
>>> To post to this group, send email to sqlalchemy@googlegroups.com
>>> <mailto:sqlalchemy@googlegroups.com>.
>>> Visit this group at https://groups.google.com/group/sqlalchemy
>>> <https://groups.google.com/group/sqlalchemy>.
>>> For more options, visit https://groups.google.com/d/optout
>>> <https://groups.google.com/d/optout>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to sqlalchemy+unsubscr...@googlegroups.com
>> <mailto:sqlalchemy+unsubscr...@googlegroups.com>.
>> To post to this group, send email to sqlalchemy@googlegroups.com
>> <mailto:sqlalchemy@googlegroups.com>.
>> Visit this group at https://groups.google.com/group/sqlalchemy
>> <https://groups.google.com/group/sqlalchemy>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
