Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-27 Thread Martijn van Oosterhout 
On Thursday, June 23, 2016 at 11:22:32 PM UTC+2, Chung wrote:
>
> Ah thanks Mike; that makes sense.
>
> What we have is actually hundreds of different databases with roughly the 
> same schema, with new databases getting created all the time live.  The 
> table schemas for each database are also being altered live.  So I'm not 
> sure we can get away with not reflecting the table schema per request; any 
> request may be referencing any new database or any newly created column. 
>  Possibly there's still something meaningful we can cache?  We will think 
> about this.
>
>
That actually sounds like a situation we had. I think that if you attach 
the connection/bind to the Session instead of to the Metadata, you can use 
the same Metadata for several different databases, as long as they have the 
same schema. Unfortunately, if you didn't set it up that way in the 
beginning it can be painful to rework...

-- 
Martijn van Oosterhout

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-23 Thread Mike Bayer 



On 06/23/2016 05:22 PM, Chung wrote:

Ah thanks Mike; that makes sense.

What we have is actually hundreds of different databases with roughly
the same schema, with new databases getting created all the time live.
 The table schemas for each database are also being altered live.  So
I'm not sure we can get away with not reflecting the table schema per
request; any request may be referencing any new database or any newly
created column.  Possibly there's still something meaningful we can
cache?  We will think about this.

Still, it is surprising that there's a concurrency conflict here.  It
looks like _mapper_registry is only added to by
Mapper._configure_class_instrumentation(), which is only called once the
_CONFIGURE_MUTEX has been acquired...


oh, so it is. So yes, it's not the addition of the mapper.   Well, it's 
also a weak-referencing dictionary, so a gc operation can occur as the 
list is being iterated, which would cause the list to change size within 
the process.






Chung

On Thursday, June 23, 2016 at 1:26:38 PM UTC-7, Mike Bayer wrote:



On 06/23/2016 03:30 PM, Chung wrote:
> We're occasionally seeing this exception thrown out of
> mapper.configure_mappers when using SQLSoup in our webapp:
>
> |
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line

> 455,in__getattr__
> returnself.entity(attr)
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line

> 452,inentity
> returnself.map_to(attr,tablename=attr,schema=schema)
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line

> 363,inmap_to
> mapper_args
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line

> 172,in_class_for_table
> fork inmappr.iterate_properties:
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line

> 1819,initerate_properties
> configure_mappers()
>
>

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line

> 2677,inconfigure_mappers
> formapper inlist(_mapper_registry):
>   File"/usr/lib/python2.7/weakref.py",line 327,initerkeys
> forwr inself.data.iterkeys():
> RuntimeError:dictionary changed size during iteration
> |
>
> I saw a previous thread about this
>

>
and
> indeed we are creating new SQLSoup instances per request not per
> process.  But I also note that since then, it seems that
sqlalchemy does
> use a _CONFIGURE_MUTEX to avoid unsynchronized reference to
> _mapper_registry.
>
> Any idea what might be causing this?

adding new mappers while the configure() step is running would cause
this, so if you are building mapped classes in requests, this would
happen.

It's incredibly inefficient to create a SQLSoup mapped class on a
per-request basis, because you're not only mapping, you're also
reflecting the table structure from the database, so unless you have
potentially millions of tables I'd look to set up all the mappers you
need up front.  The automap extension (which supersedes SQLSoup) makes
this pretty simple.





>
> Thanks,
> Chung
>
> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to sqlalchemy+...@googlegroups.com 
> .
> To post to this group, send email to sqlal...@googlegroups.com

> .
> Visit this group at https://groups.google.com/group/sqlalchemy
.
> For more options, visit https://groups.google.com/d/optout
.

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed

Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-23 Thread Chung 
Ah alternatively, Mapper instances are being garbage-collected as we 
discard some of the SQLSoup instances in finished requests, leading to 
concurrent modification.

Chung

On Thursday, June 23, 2016 at 2:22:32 PM UTC-7, Chung wrote:
>
> Ah thanks Mike; that makes sense.
>
> What we have is actually hundreds of different databases with roughly the 
> same schema, with new databases getting created all the time live.  The 
> table schemas for each database are also being altered live.  So I'm not 
> sure we can get away with not reflecting the table schema per request; any 
> request may be referencing any new database or any newly created column. 
>  Possibly there's still something meaningful we can cache?  We will think 
> about this.
>
> Still, it is surprising that there's a concurrency conflict here.  It 
> looks like _mapper_registry is only added to by 
> Mapper._configure_class_instrumentation(), which is only called once the 
> _CONFIGURE_MUTEX has been acquired...
>
> Chung
>
> On Thursday, June 23, 2016 at 1:26:38 PM UTC-7, Mike Bayer wrote:
>>
>>
>>
>> On 06/23/2016 03:30 PM, Chung wrote: 
>> > We're occasionally seeing this exception thrown out of 
>> > mapper.configure_mappers when using SQLSoup in our webapp: 
>> > 
>> > | 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>>  
>>
>> > 455,in__getattr__ 
>> > returnself.entity(attr) 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>>  
>>
>> > 452,inentity 
>> > returnself.map_to(attr,tablename=attr,schema=schema) 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>>  
>>
>> > 363,inmap_to 
>> > mapper_args 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>>  
>>
>> > 172,in_class_for_table 
>> > fork inmappr.iterate_properties: 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
>>  
>>
>> > 1819,initerate_properties 
>> > configure_mappers() 
>> > 
>> > 
>> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
>>  
>>
>> > 2677,inconfigure_mappers 
>> > formapper inlist(_mapper_registry): 
>> >   File"/usr/lib/python2.7/weakref.py",line 327,initerkeys 
>> > forwr inself.data.iterkeys(): 
>> > RuntimeError:dictionary changed size during iteration 
>> > | 
>> > 
>> > I saw a previous thread about this 
>> > <
>> https://groups.google.com/forum/#!searchin/sqlalchemy/%22RuntimeError$3A$20dictionary$20changed$20size$20during$20iteration%22/sqlalchemy/Qcwzn_GqUdQ/7n28bZqfPKYJ>
>>  
>> and 
>> > indeed we are creating new SQLSoup instances per request not per 
>> > process.  But I also note that since then, it seems that sqlalchemy 
>> does 
>> > use a _CONFIGURE_MUTEX to avoid unsynchronized reference to 
>> > _mapper_registry. 
>> > 
>> > Any idea what might be causing this? 
>>
>> adding new mappers while the configure() step is running would cause 
>> this, so if you are building mapped classes in requests, this would 
>> happen. 
>>
>> It's incredibly inefficient to create a SQLSoup mapped class on a 
>> per-request basis, because you're not only mapping, you're also 
>> reflecting the table structure from the database, so unless you have 
>> potentially millions of tables I'd look to set up all the mappers you 
>> need up front.  The automap extension (which supersedes SQLSoup) makes 
>> this pretty simple. 
>>
>>
>>
>>
>>
>> > 
>> > Thanks, 
>> > Chung 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> > Groups "sqlalchemy" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> > an email to sqlalchemy+...@googlegroups.com 
>> > . 
>> > To post to this group, send email to sqlal...@googlegroups.com 
>> > . 
>> > Visit this group at https://groups.google.com/group/sqlalchemy. 
>> > For more options, visit https://groups.google.com/d/optout. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-23 Thread Chung 
Ah thanks Mike; that makes sense.

What we have is actually hundreds of different databases with roughly the 
same schema, with new databases getting created all the time live.  The 
table schemas for each database are also being altered live.  So I'm not 
sure we can get away with not reflecting the table schema per request; any 
request may be referencing any new database or any newly created column. 
 Possibly there's still something meaningful we can cache?  We will think 
about this.

Still, it is surprising that there's a concurrency conflict here.  It looks 
like _mapper_registry is only added to by 
Mapper._configure_class_instrumentation(), which is only called once the 
_CONFIGURE_MUTEX has been acquired...

Chung

On Thursday, June 23, 2016 at 1:26:38 PM UTC-7, Mike Bayer wrote:
>
>
>
> On 06/23/2016 03:30 PM, Chung wrote: 
> > We're occasionally seeing this exception thrown out of 
> > mapper.configure_mappers when using SQLSoup in our webapp: 
> > 
> > | 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>  
>
> > 455,in__getattr__ 
> > returnself.entity(attr) 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>  
>
> > 452,inentity 
> > returnself.map_to(attr,tablename=attr,schema=schema) 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>  
>
> > 363,inmap_to 
> > mapper_args 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
>  
>
> > 172,in_class_for_table 
> > fork inmappr.iterate_properties: 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
>  
>
> > 1819,initerate_properties 
> > configure_mappers() 
> > 
> > 
> File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
>  
>
> > 2677,inconfigure_mappers 
> > formapper inlist(_mapper_registry): 
> >   File"/usr/lib/python2.7/weakref.py",line 327,initerkeys 
> > forwr inself.data.iterkeys(): 
> > RuntimeError:dictionary changed size during iteration 
> > | 
> > 
> > I saw a previous thread about this 
> > <
> https://groups.google.com/forum/#!searchin/sqlalchemy/%22RuntimeError$3A$20dictionary$20changed$20size$20during$20iteration%22/sqlalchemy/Qcwzn_GqUdQ/7n28bZqfPKYJ>
>  
> and 
> > indeed we are creating new SQLSoup instances per request not per 
> > process.  But I also note that since then, it seems that sqlalchemy does 
> > use a _CONFIGURE_MUTEX to avoid unsynchronized reference to 
> > _mapper_registry. 
> > 
> > Any idea what might be causing this? 
>
> adding new mappers while the configure() step is running would cause 
> this, so if you are building mapped classes in requests, this would 
> happen. 
>
> It's incredibly inefficient to create a SQLSoup mapped class on a 
> per-request basis, because you're not only mapping, you're also 
> reflecting the table structure from the database, so unless you have 
> potentially millions of tables I'd look to set up all the mappers you 
> need up front.  The automap extension (which supersedes SQLSoup) makes 
> this pretty simple. 
>
>
>
>
>
> > 
> > Thanks, 
> > Chung 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to sqlalchemy+...@googlegroups.com  
> > . 
> > To post to this group, send email to sqlal...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-23 Thread Mike Bayer 



On 06/23/2016 03:30 PM, Chung wrote:

We're occasionally seeing this exception thrown out of
mapper.configure_mappers when using SQLSoup in our webapp:

|

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
455,in__getattr__
returnself.entity(attr)

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
452,inentity
returnself.map_to(attr,tablename=attr,schema=schema)

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
363,inmap_to
mapper_args

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py",line
172,in_class_for_table
fork inmappr.iterate_properties:

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
1819,initerate_properties
configure_mappers()

File"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py",line
2677,inconfigure_mappers
formapper inlist(_mapper_registry):
  File"/usr/lib/python2.7/weakref.py",line 327,initerkeys
forwr inself.data.iterkeys():
RuntimeError:dictionary changed size during iteration
|

I saw a previous thread about this

 and
indeed we are creating new SQLSoup instances per request not per
process.  But I also note that since then, it seems that sqlalchemy does
use a _CONFIGURE_MUTEX to avoid unsynchronized reference to
_mapper_registry.

Any idea what might be causing this?


adding new mappers while the configure() step is running would cause 
this, so if you are building mapped classes in requests, this would happen.


It's incredibly inefficient to create a SQLSoup mapped class on a 
per-request basis, because you're not only mapping, you're also 
reflecting the table structure from the database, so unless you have 
potentially millions of tables I'd look to set up all the mappers you 
need up front.  The automap extension (which supersedes SQLSoup) makes 
this pretty simple.








Thanks,
Chung

--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
.
To post to this group, send email to sqlalchemy@googlegroups.com
.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] "RuntimeError: dictionary changed size during iteration" during configure_mappers

2016-06-23 Thread Chung 
We're occasionally seeing this exception thrown out of 
mapper.configure_mappers when using SQLSoup in our webapp:

  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py"
, line 455, in __getattr__
return self.entity(attr)
  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py"
, line 452, in entity
return self.map_to(attr, tablename=attr, schema=schema)
  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py"
, line 363, in map_to
mapper_args
  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlsoup.py"
, line 172, in _class_for_table
for k in mappr.iterate_properties:
  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py"
, line 1819, in iterate_properties
configure_mappers()
  File 
"/home/glados/.virtualenvs/pod-glance/local/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py"
, line 2677, in configure_mappers
for mapper in list(_mapper_registry):
  File "/usr/lib/python2.7/weakref.py", line 327, in iterkeys
for wr in self.data.iterkeys():
RuntimeError: dictionary changed size during iteration

I saw a previous thread about this 

 and 
indeed we are creating new SQLSoup instances per request not per process. 
 But I also note that since then, it seems that sqlalchemy does use 
a _CONFIGURE_MUTEX to avoid unsynchronized reference to _mapper_registry.

Any idea what might be causing this?

Thanks,
Chung

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.