[sqlalchemy] Re: 'lazy=False' with polymorphic joined tables
Michael Bayer wrote: i havent looked yet but this is likely a bug in eager loading when it interacts with polymorphic. are you on trunk ? Nope. I just installed SQLAlchemy-0.4.2dev_r3952. And it's now working. SA 0.4.1: [...] FROM companies JOIN suppliers ON companies.id = suppliers.id JOIN addresses ON addresses.id = companies.id_address, companies JOIN suppliers ON companies.id = suppliers.id LEFT OUTER JOIN addresses AS addresses_1 ON addresses_1.id = companies.id_address [...] SA 0.4.2dev_r3952: [...] FROM companies INNER JOIN suppliers ON companies.id = suppliers.id INNER JOIN addresses ON addresses.id = companies.id_address LEFT OUTER JOIN addresses AS addresses_1 ON addresses_1.id = companies.id_address [...] Sorry, I forgot to check the trunk before sending to the list... Regards, -- Alexandre CONRAD - TLV FRANCE Research Development --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Firebird status and some testsuite issues
Hi all,
I was finally able to spend a little time on the Firebird backend, and
I'm glad to say that I'm currently down to this test summary:
Ran 1030 tests in 63.169s
FAILED (failures=7, errors=187)
There are still some real issue, but most of the failures and errors
come from a few common patterns:
A. some tests do a textual comparison against an expected
statement; this breaks on FB, because the dialect inserts an
explicit PK field::
AssertionError: Testing for query 'INSERT INTO users (user_name)
VALUES (?)' params [{'user_name': 'thesub'}], received
'INSERT INTO users (user_id, user_name) VALUES (?, ?)'
with params [{'user_name': 'thesub', 'user_id': 1}]
five of the above seven failures are of this kind :)
B. many other tests insert an explicit NULL in a PK field, not
allowed under Firebird (because PK fields *must* be NOT NULL)
most of the above 187 errors fall in this category.
C. a few tests don't use an explicit ordering, and assume that the
result of a select without an ORDER BY matches the insert order;
under Firebird I notice an intermittent behaviour, and in some
cases (expecially under load) this is not true. The following
fixes one of those points to explain::
Index: test/engine/execute.py
===
--- test/engine/execute.py (revisione 3952)
+++ test/engine/execute.py (copia locale)
@@ -26,7 +26,7 @@
conn.execute(insert into users (user_id, user_name) values
(?, ?), [3,ed], [4,horse])
conn.execute(insert into users (user_id, user_name) values
(?, ?), (5,barney), (6,donkey))
conn.execute(insert into users (user_id, user_name) values
(?, ?), 7, 'sally')
-res = conn.execute(select * from users)
+res = conn.execute(select * from users order by user_id)
assert res.fetchall() == [(1, jack), (2, fred), (3,
ed), (4, horse), (5, barney), (6, donkey), (7, 'sally')]
conn.execute(delete from users)
D. to my surprise, a self-reference foreign key on a table needs
either ON DELETE CASCADE or ON DELETE SET NULL to allow the
testsuite to delete its content in the tearDown step with a simple
DELETE FROM table statement; one spot of this is fixed by the
following::
Index: test/orm/eager_relations.py
===
--- test/orm/eager_relations.py (revisione 3952)
+++ test/orm/eager_relations.py (copia locale)
@@ -650,7 +650,7 @@
global nodes
nodes = Table('nodes', metadata,
Column('id', Integer, Sequence('node_id_seq',
optional=True), primary_key=True),
-Column('parent_id', Integer, ForeignKey('nodes.id')),
+Column('parent_id', Integer, ForeignKey('nodes.id',
onupdate=CASCADE, ondelete=CASCADE)),
Column('data', String(30)))
@testing.fails_on('maxdb')
I don't care so much about As because it's clearly an annoying
problem to cure: hopefully those stmts are going to be exercised for
real...
Bs annoy too, and void the effectiveness of several tests.
I'd like to correct the last two points, because otherwise important
tests fail for different reasons than the test expects to trigger,
masquerading real issues. For Ds, I understand that not all DBs handle
that kind of FK so maybe those backend could ignore
onupdate/ondelete?
What do you think?
Thank you,
ciao, lele.
--
nickname: Lele Gaifax| Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas| comincerò ad aver paura di chi mi copia.
[EMAIL PROTECTED] | -- Fortunato Depero, 1929.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Using ORM Object as an intermadiate holder
On 16 Aralık, 22:22, Michael Bayer [EMAIL PROTECTED] wrote: On Dec 15, 11:02 pm, Utku Altinkaya [EMAIL PROTECTED] wrote: I do not want to lose invalid values, becouse I want to send them to the user again, so while using object as intermediate holder I have to set attributes invalid values... So Autoflush = False, and if invalid values are existed the object is reloaded from DB before commit step of web request cycle. But while using SQLAlchemy I had the impression that it is not designed to to that, It designers thought objects are direct representation of the data in the database. I can do this ofcouse some kind of holder class copies attributes from data objects etc, but it is cumbersome, and will force me to write longer code. I am expecting someone to tell me the way I am doing is perfectly valid, or another advice the usual way people handle form validation is using a package like FormEncode: http://formencode.org/Validator.html OK, I have started implementing widgets to do this, I would do things like that some place anyway, I will not use data objects for self validation etc anymore. i.e. you wouldn't have the invalid data on your ORM object at any point. theres nothing wrong with how you're doing it, it just has disadvantages. namely, that your ORM objects have to look just like your web forms (such as, a web form with three dropdowns, month, day and year... Actullay, I was using mako templates, and setting a context variable to the model(single thread for a single template instance ofcourse), and using some of my helper components to write the html, so converting data to the representable form to the user via html form had been done there. It was short and simple, I liked it. but your ORM object only has date on it), and you also have to ensure the ORM objects dont get flushed with the invalid data (easily breakable). And looks bad, expriding the object I mean, thanks for the advice. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Fwd: [elixir] warnings on exit
On Dec 13, 5:49 pm, Michael Bayer [EMAIL PROTECTED] wrote: these are some cleanup messages which occur during the final gc of objects in the session. ive tried to work around them a bit but they still persist for some applications ( i havent yet been able to reproduce them and im hesitatnt to just squash all the exceptions entirely). they are harmless, but also can be prevented if you close() the session explicily before the app completes. No, it does not: closing the session was the first thing I tried. In my particular case the warnings *are* somewhat harmful: they make it a lot more difficult to read the effects of running the unit tests for my program. Of course, I can workaround it by sending stderror to stdoutput and piping it to less, but I am not completely happy about this solution. (Right now I am out of my office and I don't have access to my code or sqlalchemy version I am using... When I get there on Tuesday I will double check if closing the session doesn't help.) Best regards, -- Richard --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Problem with Relashionship
Please people help me
My Simple Model
class Record:
has_field('special_number',Unicode(100))
belongs_to('person',of_kind='Person')
class Person:
has_field('name',Unicode(100))
=
Use this:
Record(special_number=111,person=Person(name='myname'))
But now a i need access the special_number attribute in Record
starting at Person
I try this and don't work:
print Person.query(Record).one().special_number
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Hi to ALL. I just join this group.
Greetings to all http://milearmida.tripod.com/widescreen-wallpapers-naruto.html widescreen wallpapers naruto http://milearmida.tripod.com/index.html naruto wallpapers http://milearmida.tripod.com/naruto-akatsuki-wallpapers.html naruto akatsuki wallpapers http://milearmida.tripod.com/naruto-movie-1-wallpapers.html naruto movie 1 wallpapers http://milearmida.tripod.com/black-and-white-naruto-wallpapers.html black and white naruto wallpapers --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Many-To-One...What I'm doing wrong?
That works. Thank's a lot.
Marcos.
I will continue asking. Soon...
On Dec 14, 7:20 pm, [EMAIL PROTECTED] wrote:
your 'iva' table-column AND 'iva' attribute/relation/property have same
name, Thats what the error says. either rename one of them (e.g. the column
to become iva_id), or use that allow_column_override=True flag to the
producto mapper.
Marcos wrote:
Hello, first at all, sorry about my english... I'm from Mendoza,
Argentina, and I'm starting with this. Because I can't describe my
problem, I will show the next:
from sqlalchemy import *
from sqlalchemy.orm import *
class Producto(object):
pass
class Iva(object):
pass
metadata = MetaData()
engine = create_engine('postgres://postgres:[EMAIL PROTECTED]:5432/
marcos')
connection = engine.connect()
Session = sessionmaker(bind=connection, autoflush=True,
transactional=True)
session = Session()
iva_tabla = Table( 'iva' , metadata ,
Column('id' , Integer , primary_key =
True) ,
Column('valor', Float)
)
productos_tabla = Table ( 'productos' , metadata ,
Column('id', Integer ,
primary_key = True ) ,
Column('nro' , Integer ) ,
Column('descripcion' ,
String(100) ) ,
Column('iva' , Integer ,
ForeignKey('iva.id')) ,
Column('precioUnitario' ,
Float )
)
mapper(Producto, productos_tabla , properties={'iva':relation(Iva)})
mapper(Iva,iva_tabla)
- And when i run it:
raceback (most recent call last):
File /home/instancia_webware/MyContext/Pruebas/sql.py, line 29,
in ?
mapper(Producto, productos_tabla ,
properties={'iva':relation(Iva)})
File /usr/lib/python2.4/site-packages/SQLAlchemy-0.4.1-py2.4.egg/
sqlalchemy/orm/__init__.py, line 518, in mapper
return Mapper(class_, local_table, *args, **params)
File /usr/lib/python2.4/site-packages/SQLAlchemy-0.4.1-py2.4.egg/
sqlalchemy/orm/mapper.py, line 154, in __init__
self._compile_properties()
File /usr/lib/python2.4/site-packages/SQLAlchemy-0.4.1-py2.4.egg/
sqlalchemy/orm/mapper.py, line 612, in _compile_properties
self._compile_property(column_key, column, init=False,
setparent=True)
File /usr/lib/python2.4/site-packages/SQLAlchemy-0.4.1-py2.4.egg/
sqlalchemy/orm/mapper.py, line 654, in _compile_property
raise exceptions.ArgumentError(WARNING: column '%s' not being
added due to property '%s'. Specify 'allow_column_override=True' to
mapper() to ignore this condition. % (column.key, repr(prop)))
sqlalchemy.exceptions.ArgumentError: WARNING: column 'iva' not being
added due to property 'sqlalchemy.orm.properties.PropertyLoader
object at 0xb7aba1cc'. Specify 'allow_column_override=True' to
mapper() to ignore this condition.
--
I've read a lot, and I can't or I'm incapable to resolve this by
myself. Can anybody help me?
Sorry again, Marcos
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Python in instead of list of or_ values.
On Dec 17, 2007, at 11:32 AM, Justin wrote: Using the Django ORM you can write: MyModel.objects.filter(property__in=['list', 'of', 'values']) ...and It will OR those values for you. Does SQLAlchemy have a similar shortcut? table.c.somecol.in_([list of values]) MyClass.someattribute.in_([list of values]) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Fwd: [elixir] warnings on exit
On Dec 16, 2007, at 7:29 PM, Ryszard Szopa wrote: On Dec 13, 5:49 pm, Michael Bayer [EMAIL PROTECTED] wrote: these are some cleanup messages which occur during the final gc of objects in the session. ive tried to work around them a bit but they still persist for some applications ( i havent yet been able to reproduce them and im hesitatnt to just squash all the exceptions entirely). they are harmless, but also can be prevented if you close() the session explicily before the app completes. No, it does not: closing the session was the first thing I tried. In my particular case the warnings *are* somewhat harmful: they make it a lot more difficult to read the effects of running the unit tests for my program. Make sure you are closing all sessions that are created. Also, its important that you show us what the exact warnings youre getting are. Please send along a reproducing test case illustrating the full issue. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Firebird status and some testsuite issues
On Dec 17, 2007, at 8:59 AM, Lele Gaifax wrote:
A. some tests do a textual comparison against an expected
statement; this breaks on FB, because the dialect inserts an
explicit PK field::
AssertionError: Testing for query 'INSERT INTO users (user_name)
VALUES (?)' params [{'user_name': 'thesub'}], received
'INSERT INTO users (user_id, user_name) VALUES (?, ?)'
with params [{'user_name': 'thesub', 'user_id': 1}]
five of the above seven failures are of this kind :)
there are two varieties to those types of test, one with sequence and
one without. you shuld change line 433 of test/testlib/testing.py to
include firebird in the list of dbs that use sequences.
B. many other tests insert an explicit NULL in a PK field, not
allowed under Firebird (because PK fields *must* be NOT NULL)
most of the above 187 errors fall in this category.
all those tests must be given a Sequence object with the
optional=True flag on the first integer primary key column, and
Firebird must use sequences that are marked as optional (since they
arent optional with FB). One way we can do this in an automated
fashion would be to complete the Column implementation in test/testlib/
schema.py to automatically gen a Sequence for primary-key integer cols
that do not already have a sequence (though this might have side
effects for some tests).
C. a few tests don't use an explicit ordering, and assume that the
result of a select without an ORDER BY matches the insert order;
under Firebird I notice an intermittent behaviour, and in some
cases (expecially under load) this is not true. The following
fixes one of those points to explain::
Index: test/engine/execute.py
===
--- test/engine/execute.py (revisione 3952)
+++ test/engine/execute.py (copia locale)
@@ -26,7 +26,7 @@
conn.execute(insert into users (user_id,
user_name) values (?, ?), [3,ed], [4,horse])
conn.execute(insert into users (user_id,
user_name) values (?, ?), (5,barney), (6,donkey))
conn.execute(insert into users (user_id,
user_name) values (?, ?), 7, 'sally')
-res = conn.execute(select * from users)
+res = conn.execute(select * from users order by
user_id)
assert res.fetchall() == [(1, jack), (2,
fred), (3, ed), (4, horse), (5, barney), (6, donkey), (7,
'sally')]
conn.execute(delete from users)
explicit orderings can be added as needed to suit result set
comparisons; we have to add these all the time for postgres.
D. to my surprise, a self-reference foreign key on a table needs
either ON DELETE CASCADE or ON DELETE SET NULL to allow the
testsuite to delete its content in the tearDown step with a simple
DELETE FROM table statement; one spot of this is fixed by the
following::
Index: test/orm/eager_relations.py
===
--- test/orm/eager_relations.py (revisione 3952)
+++ test/orm/eager_relations.py (copia locale)
@@ -650,7 +650,7 @@
global nodes
nodes = Table('nodes', metadata,
Column('id', Integer, Sequence('node_id_seq',
optional=True), primary_key=True),
-Column('parent_id', Integer,
ForeignKey('nodes.id')),
+Column('parent_id', Integer,
ForeignKey('nodes.id', onupdate=CASCADE, ondelete=CASCADE)),
Column('data', String(30)))
I was wondering the other day if any DB's would have this issue. Im
not comfortable adding CASCADE rules in every case, although in
eager_relations is probably OK, since sometimes we are testing that
SQLA explicitly deletes things properly. I think for this we might
look at adding a ForeignKey construct to test/testlib/schema.py which
adds the CASCADE rules for the firebird dialect only.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Python in instead of list of or_ values.
Thanks! - Justin On Dec 17, 2007 11:44 AM, Michael Bayer [EMAIL PROTECTED] wrote: On Dec 17, 2007, at 11:32 AM, Justin wrote: Using the Django ORM you can write: MyModel.objects.filter(property__in=['list', 'of', 'values']) ...and It will OR those values for you. Does SQLAlchemy have a similar shortcut? table.c.somecol.in_([list of values]) MyClass.someattribute.in_([list of values]) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: obtaining previous value in mapper.extension.after_*
On Dec 16, 2007, at 3:26 PM, [EMAIL PROTECTED] wrote: and another issue around attribute.get_history... i have a descriptor that is autosetting some defaultvalue at first get. a descriptor on top of the InstrumentedAttribute itself ? id wonder how you are configuring that. ScalarObjectAttributeImpl never knows that the attribute has been missing at start - dict.get(key,NOVALUE) will never return NOVALUE, as the descriptor machinery is called instead of __dict__[key] / haskey etc. if you really want a default on first get value, and youre riding directly on top of the IA instead of using a differently-named attribute for your own descriptor like all the docs say to do, you should use the callable mechanisms built into the attributes package, although the new value created becomes the committed state so maybe thats not what you want. im confused, someone issues print object.foo, then it inserts some new data instead of not doing anything ? what if nobody gets the attribute ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: obtaining previous value in mapper.extension.after_*
Michael Bayer wrote: On Dec 16, 2007, at 3:26 PM, [EMAIL PROTECTED] wrote: and another issue around attribute.get_history... i have a descriptor that is autosetting some defaultvalue at first get. a descriptor on top of the InstrumentedAttribute itself ? id wonder how you are configuring that. under IA. in the __dict__ (which is not a dict at all). ScalarObjectAttributeImpl never knows that the attribute has been missing at start - dict.get(key,NOVALUE) will never return NOVALUE, as the descriptor machinery is called instead of __dict__[key] / haskey etc. if you really want a default on first get value, and youre riding directly on top of the IA instead of using a differently-named attribute for your own descriptor like all the docs say to do, yes and no, as i said i'm replacing the __dict__ with something special; so its IA riding on top of me (;-) but otherwise its that. no renaming, i dont want someone (thats can be me, later) to be able to workaround either me or SA. you should use the callable mechanisms built into the attributes package, although the new value created becomes the committed state so maybe thats not what you want. im confused, someone issues print object.foo, then it inserts some new data instead of not doing anything ? what if nobody gets the attribute ? then it remains unset (None, null, whatever). and i have a.b.c.d = 3 where b and c are auto-created (if declared so of course) as for autoinsert -- what people write is what people get. i know it may be confusing; its a sort of declarative attribute-type metainfo, with validation, conversion, readonly, autoset, default-value, optional'ism, hints about UI/str/whatever representation, comment etc etc). And static-typing/static-attribute-set of course. db-business objects should not have any random stuff hanging on them. trouble is it partialy duplicates some SA features so there is a conflict of interests, since day one of their meeting. i was thinking about splitting it into two and putting one part above IA using those renamed columns approach and the rest underneath but that would be lots of work. later svilen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: obtaining previous value in mapper.extension.after_*
yes and no, as i said i'm replacing the __dict__ with something special; so its IA riding on top of me (;-) but otherwise its that. no renaming, i dont want someone (thats can be me, later) to be able to workaround either me or SA. then have your magic __dict__ implement the same save committed on change behavior as the attributes package. and of course test on every SA release (i know, it broke only 6 times this year ;-) since youre welded to internal behavior: def my_magic_dict_set_something(dict, key, value): if key not in dict['_state'].committed_state: dict['_state'].committed_state[key] = _the_old_value dict[key] = value thanks, thats different idea to what i had in mind... i'll try.. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] traversal order of ClauseVistor
Hi, I am just wondering about the traversal order of ClauseVistor, does it have to be in a set order? I am eagerloading lots (1000s) of relations, and query compilation take a long time, a profiling revealed that most of the time was spent in traverse. So, I am just wondering if the performance will bump up if we do not have to use intermediate lists (or use one less list) to maintain the traversal order. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
On Dec 17, 2007, at 10:38 PM, Esceo wrote: Hi, I am just wondering about the traversal order of ClauseVistor, does it have to be in a set order? I am eagerloading lots (1000s) of relations, and query compilation take a long time, a profiling revealed that most of the time was spent in traverse. it sounds like you are on an old version of SQLAlchemy as the compiler does not use any method called traverse anymore. If you are on any 0.4 version, our statement compilation is quite fast; as of 0.4.1 INSERTs and UPDATEs take around 40-50 function calls and a SELECT about 120-150 (the callcount goes up with complexity obviously). Also if you can clarify what eagerloading 1000s of relations means, since i doubt you are issuing a JOIN of 1000 tables, that would be helpful. If youre on an 0.3 version and switch to 0.4.1 youll see a significant decrease in processing time for just about everything, and 0.4.2 is going to improve an average of about 20% over 0.4.1 (available in trunk if you feel like trying it). So, I am just wondering if the performance will bump up if we do not have to use intermediate lists (or use one less list) to maintain the traversal order. Again assuming 0.4, there's no lists of traversal order used in statement compilation, it calls each function as needed in order to render each component of the select, using regular recursive generation, and the order of function calls is exactly the order of each component in the final string output. We profile all the time, in fact on a daily basis, and we are quite fast; if you've seen those infamous Robert Brewer tests, we actually beat out at least one of the competitors when the tests are made fair by removing the hundreds of transaction commits that only the SQLAlchemy test, but not the others, is made to do. I'm debating doing some more blog posts about that. If youre talking about the visitiors.py traversal, thats not used right now in statement compilation, and the 0.4 version is also an inlined, non-recursive version which is generally faster than a recursive algorithm since it incurs very little method call overhead, which tends to be one of the most expensive things in Python. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
Hi Michael, Thanks for the quick reply. What I am in fact doing is undefer all fields, eagerload all relations (and undefer the corresponding fields) for a corresponding model. (ended up being 9xx options altogether) And yes, I am on 0.3.11 (and probably is unable to move onto a 0.4 as I am relying on an old and custom version of elixir and some custom sqlalchemy fixes), I guess the question is whether it is possible (easy) to rip the benefit of those improvements by merging the 0.4 code into my current 0.3.11? Lei I guess the question now is, On Dec 18, 4:25 pm, Michael Bayer [EMAIL PROTECTED] wrote: On Dec 17, 2007, at 10:38 PM, Esceo wrote: Hi, I am just wondering about the traversal order of ClauseVistor, does it have to be in a set order? I am eagerloading lots (1000s) of relations, and query compilation take a long time, a profiling revealed that most of the time was spent in traverse. it sounds like you are on an old version of SQLAlchemy as the compiler does not use any method called traverse anymore. If you are on any 0.4 version, our statement compilation is quite fast; as of 0.4.1 INSERTs and UPDATEs take around 40-50 function calls and a SELECT about 120-150 (the callcount goes up with complexity obviously). Also if you can clarify what eagerloading 1000s of relations means, since i doubt you are issuing a JOIN of 1000 tables, that would be helpful. If youre on an 0.3 version and switch to 0.4.1 youll see a significant decrease in processing time for just about everything, and 0.4.2 is going to improve an average of about 20% over 0.4.1 (available in trunk if you feel like trying it). So, I am just wondering if the performance will bump up if we do not have to use intermediate lists (or use one less list) to maintain the traversal order. Again assuming 0.4, there's no lists of traversal order used in statement compilation, it calls each function as needed in order to render each component of the select, using regular recursive generation, and the order of function calls is exactly the order of each component in the final string output. We profile all the time, in fact on a daily basis, and we are quite fast; if you've seen those infamous Robert Brewer tests, we actually beat out at least one of the competitors when the tests are made fair by removing the hundreds of transaction commits that only the SQLAlchemy test, but not the others, is made to do. I'm debating doing some more blog posts about that. If youre talking about the visitiors.py traversal, thats not used right now in statement compilation, and the 0.4 version is also an inlined, non-recursive version which is generally faster than a recursive algorithm since it incurs very little method call overhead, which tends to be one of the most expensive things in Python. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
Hi,
Seeing so many improvements and benefits, I might as well migrate to
0.4
Just few things I wanted to make sure before I start on that.
Inside the 0.4 branch,
1) are we still generating anonymous labels with a width of 4 char
('anon_0fda') etc?
2) there is a bug inside 0.3's strategies deferred column loader (in
maintenance branch as well), basically, when the parent's primary key
happens to be 0, deferred loading is faulty (if not attr: should have
been if not attr == None: around line 94 in strategies.py), has this
been fixed?
3) another bug in 0.3's InstrumentedAttribute system where if we set
the same value to an instrumented attribute twice, the value's parent
will be reset (to no parent)
(in resetting old's parent, we didn't check if old == value, around
line 269 in attributes.py)
4) the other thing is, property loader is ignoring it's own attribute
extensions when there is a backref, and will use the backref's
attribute extensions instead, is that meant to be the case? (same
thing in 0.4?)
Thanks in Advance
Lei
On Dec 18, 4:57 pm, Esceo [EMAIL PROTECTED] wrote:
Hi Michael,
Thanks for the quick reply.
What I am in fact doing is undefer all fields, eagerload all relations
(and undefer the corresponding fields) for a corresponding model.
(ended up being 9xx options altogether)
And yes, I am on 0.3.11 (and probably is unable to move onto a 0.4 as
I am relying on an old and custom version of elixir and some custom
sqlalchemy fixes),
I guess the question is whether it is possible (easy) to rip the
benefit of those improvements by merging the 0.4 code into my current
0.3.11?
Lei
I guess the question now is,
On Dec 18, 4:25 pm, Michael Bayer [EMAIL PROTECTED] wrote:
On Dec 17, 2007, at 10:38 PM, Esceo wrote:
Hi,
I am just wondering about the traversal order of ClauseVistor, does it
have to be in a set order?
I am eagerloading lots (1000s) of relations, and query compilation
take a long time, a profiling revealed that most of the time was spent
in traverse.
it sounds like you are on an old version of SQLAlchemy as the compiler
does not use any method called traverse anymore. If you are on any
0.4 version, our statement compilation is quite fast; as of 0.4.1
INSERTs and UPDATEs take around 40-50 function calls and a SELECT
about 120-150 (the callcount goes up with complexity obviously). Also
if you can clarify what eagerloading 1000s of relations means, since
i doubt you are issuing a JOIN of 1000 tables, that would be helpful.
If youre on an 0.3 version and switch to 0.4.1 youll see a significant
decrease in processing time for just about everything, and 0.4.2 is
going to improve an average of about 20% over 0.4.1 (available in
trunk if you feel like trying it).
So, I am just wondering if the performance will bump up
if we do not have to use intermediate lists (or use one less list) to
maintain the traversal order.
Again assuming 0.4, there's no lists of traversal order used in
statement compilation, it calls each function as needed in order to
render each component of the select, using regular recursive
generation, and the order of function calls is exactly the order of
each component in the final string output. We profile all the time,
in fact on a daily basis, and we are quite fast; if you've seen those
infamous Robert Brewer tests, we actually beat out at least one of the
competitors when the tests are made fair by removing the hundreds of
transaction commits that only the SQLAlchemy test, but not the others,
is made to do. I'm debating doing some more blog posts about that.
If youre talking about the visitiors.py traversal, thats not used
right now in statement compilation, and the 0.4 version is also an
inlined, non-recursive version which is generally faster than a
recursive algorithm since it incurs very little method call overhead,
which tends to be one of the most expensive things in Python.- Hide quoted
text -
- Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
On Dec 18, 2007, at 12:57 AM, Esceo wrote: Hi Michael, Thanks for the quick reply. What I am in fact doing is undefer all fields, eagerload all relations (and undefer the corresponding fields) for a corresponding model. (ended up being 9xx options altogether) And yes, I am on 0.3.11 (and probably is unable to move onto a 0.4 as I am relying on an old and custom version of elixir and some custom sqlalchemy fixes), I guess the question is whether it is possible (easy) to rip the benefit of those improvements by merging the 0.4 code into my current 0.3.11? no, 0.4 is almost a total rewrite of 0.3's internals.Of course its possible since its been done, but then youd just be left with... 0.4! Which you can just download. I'm not sure how extensive your elixir/SA hacks are but the Elixir project has also been improving and updating, so if your application is reasonable you should be able to upgrade and do away with the hacks. And if youve fixed SQLAlchemy bugs I would certainiy hope that you've posted trac tickets so they've been fixed here as well. This is all release early, release often open source software, all in relatively low version numbers, so coding to a private fork is not generally not a good idea. If 9xx means900, im not exactly sure what you hope to accomplish thereeagerloads shouldnt be used across more than half a dozen tables or so, else the efficiency of the database's optimizer, not to mention the enormous explosion of rows youd get back, will grind any system to a halt. So a number of like 10 eagerloading attributes is already counterproductive.For a number in the hundreds I'd think your system would run out of RAM just trying to generate the query, not to mention passing it to the database's query parser, so that sounds a little strange. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
no, 0.4 is almost a total rewrite of 0.3's internals.Of course its possible since its been done, but then youd just be left with... 0.4! Which you can just download. I'm not sure how extensive your elixir/SA hacks are but the Elixir project has also been improving and updating, so if your application is reasonable you should be able to upgrade and do away with the hacks. And if youve fixed SQLAlchemy bugs I would certainiy hope that you've posted trac tickets so they've been fixed here as well. This is all release early, release often open source software, all in relatively low version numbers, so coding to a private fork is not generally not a good idea. yep, has decided to do that, and I will post tickets regarding those bugs. (just have to merge my elixir with the new elixir as well...) If 9xx means900, im not exactly sure what you hope to accomplish thereeagerloads shouldnt be used across more than half a dozen tables or so, else the efficiency of the database's optimizer, not to mention the enormous explosion of rows youd get back, will grind any system to a halt. So a number of like 10 eagerloading attributes is already counterproductive.For a number in the hundreds I'd think your system would run out of RAM just trying to generate the query, not to mention passing it to the database's query parser, so that sounds a little strange.- Hide quoted text - I am actually pulling back 900 or so fields at the same time, but it's still way faster than having to dig through the hierarchy of information and issuing separate select statement for each new aspect (perhaps because of network configuration) Anyhow, thanks very much again for the good work :) - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: traversal order of ClauseVistor
On Dec 18, 2007, at 1:34 AM, Esceo wrote:
Hi,
Seeing so many improvements and benefits, I might as well migrate to
0.4
Just few things I wanted to make sure before I start on that.
Inside the 0.4 branch,
1) are we still generating anonymous labels with a width of 4 char
('anon_0fda') etc?
we generate anonymous labels with a name, underscore, and an integer
counter now. the random hex strings are gone and the label names are
deterministic for a particular SQL expression (i.e. the same construct
produces the same string every time).
2) there is a bug inside 0.3's strategies deferred column loader (in
maintenance branch as well), basically, when the parent's primary key
happens to be 0, deferred loading is faulty (if not attr: should have
been if not attr == None: around line 94 in strategies.py), has this
been fixed?
I doubt that issue exists, but if it does its not very hard to
change. The auto PK gen of mysql and sqlite is 1-based as well as the
behavior of postgres and oracle sequences so im not sure how youre
getting zeros in the first place.
3) another bug in 0.3's InstrumentedAttribute system where if we set
the same value to an instrumented attribute twice, the value's parent
will be reset (to no parent)
(in resetting old's parent, we didn't check if old == value, around
line 269 in attributes.py)
ive never heard of that issue, and also im not really sure what you
mean by parent, are you referring to the has_parents flag ? a
short test case illustrating the behavior would be of use all around
here.
4) the other thing is, property loader is ignoring it's own attribute
extensions when there is a backref, and will use the backref's
attribute extensions instead, is that meant to be the case? (same
thing in 0.4?)
Also not sure what you mean here; the attribute extensions themselves
don't know anything about the backref concept, and PropertyLoaders
dont ignore extensions or use any of them, they just configure
them at init time and send them off to the attributes package, where
they fire off for all events for which they listen. When a backref is
used, additional extensions are added to manage the bidirectional
relationship but that has no effect on the cascade event handlers. If
you're referring to the interaction between two properties on a many-
to-many relation, its necessary that only one dependency processor
handle the updates to the association table, but outside of that,
bidirectional relations have no explicit interaction with each other
after initialization. Again, a short test script illustrating the end
behavior youre observing would be helpful here.
If you were actually building your own structures using
AttributeExtensions, I can guess that they may have behaviors not
suited to your specific use cases, but they are also not really a
public API - if you want to instrument the getting/setting of
attributes, the established way to do that is to use regular Python
properties. in 0.4.2 we have a more abbreviated syntax for creating
properties that proxy to column attributes. But if you were using AEs
for something, you'd certainly need to talk to us so that we can make
sure that they are the appropriate thing to use, or if we need to give
you some other way to support the use case you're trying to accomplish.
Also I would encourage you to get into the habit of posting bugs and
undesired behaviors as trac tickets and/or mailing list emails...this
helps both you and the project in myriad ways, by allowing us to fix
things, clarify things, identify desired use cases that should be
supported, and also to better define how we'd like the library to be
used and how we wouldn't (for example, if some part of SA which we had
intended to be private starts getting used by the public, thats a
heads up to us that we need to establish more clearly the public/
privateness of certain areas, and possibly provide a public interface
to that functionality). The SQLAlchemy software itself is worth very
little without the community effort behind it, and SA is in particular
an extremely community-built project, formed almost entirely as a
result of an ongoing dialogue with users. I think everyone knows we
respond extremely quickly to most issues so theres really no reason to
keep bugs a secret.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
