[sqlalchemy] Re: using the native c implementation of ordereddict
On Thursday 04 September 2008 15:42:56 Michael Bayer wrote:
On Sep 4, 2008, at 12:40 AM, gniquil wrote:
Hi All,
I am doing some work with xmlrpc. One thing I realize is that
whenever I pass dict(row) through xmlrpc, I get an key-ordered
struct. But this isn't what i really want. What I want is ordered
by insertion or the original list order. This led me to look at
the util.ordereddict implementation, which is pure python, which
is slow. I looked around and found this:
http://www.xs4all.nl/~anthon/Python/ordereddict/
which is a c-implementation. At the bottom of the page, there are
performance tests. It's much faster. I've got some pretty
gigantic tables to pass around, which i think this would really
help. Hopefully this could somehow find itself into next official
python. But before that, we can use this or we can just
incorporate it somehow in sqlalchemy...as a suggestion.
the problem with saying utility class X is slow, therefore use Y
is that you haven't evaluated if the slowness of X is really
impacting the performance of SQLAlchemy overall in a negative way.
I think if you ran some profiling results you'd see that
OrderedDict calls make up a miniscule portion of time spent for
doing all operations, so an external dependency is not necessarily
worth it in this case (though it may be). I have some vague
recollection that our own ODict does some things the native one
does not but I'd have to dig back into the code to remember what
they were. If our own ODict could be swappable with ordereddict,
we could at least try to import it then fall back to our own (this
is what it would look like if ordereddict were introduced into
python core anyway).
i used to set sqlalchemy.util.Set to be sqlalchemy.util.OrderedSet and
that worked well...
if all those basic things (dict, set, odict, oset, etc) are always
routed via sqlachemy.util, then one can replace them with whatever
fancy. One main usage is that testing cases would be more repeatable,
because flush plans and other hash-relating things will be same, if
all sets are ordered and all dicts are ordered.
this is not going to impact anything speedwise, it only means changing
several hundred places where {} or set() is used, and keeping some
discipline of not introducing those in future code.
someone may tell me about a way to directly hack pythons notion of {}
with something mine... would be good but is going to impact speed of
*any* code, not just SA.
mike, sorry for repeating myself again on the theme :-)
i can prepare The patch as long as u decide to keep such protocol...
svil
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: using the native c implementation of ordereddict
Michael Bayer wrote: On Sep 4, 2008, at 12:40 AM, gniquil wrote: Hi All, I am doing some work with xmlrpc. One thing I realize is that whenever I pass dict(row) through xmlrpc, I get an key-ordered struct. But this isn't what i really want. What I want is ordered by insertion or the original list order. This led me to look at the util.ordereddict implementation, which is pure python, which is slow. I looked around and found this: http://www.xs4all.nl/~anthon/Python/ordereddict/ which is a c-implementation. At the bottom of the page, there are performance tests. It's much faster. I've got some pretty gigantic tables to pass around, which i think this would really help. Hopefully this could somehow find itself into next official python. But before that, we can use this or we can just incorporate it somehow in sqlalchemy...as a suggestion. the problem with saying utility class X is slow, therefore use Y is that you haven't evaluated if the slowness of X is really impacting the performance of SQLAlchemy overall in a negative way. I think if you ran some profiling results you'd see that OrderedDict calls make up a miniscule portion of time spent for doing all operations, so an external dependency is not necessarily worth it in this case (though it may be). I have some vague recollection that our own ODict does some things the native one does not but I'd have to dig back into the code to remember what they were. If our own ODict could be swappable with ordereddict, we could at least try to import it then fall back to our own (this is what it would look like if ordereddict were introduced into python core anyway). fwiw i spiked this out a while back (just before 0.4.0, maybe), and swapping in a native ordered dict was a very marginal speed improvement, and most of it was in metadata setup rather than runtime speed. as svil said, it's easy to try this out by monkeypatching in alternate implementations and then hitting the various profiling and speed tests in the test suite. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: using the native c implementation of ordereddict
On Thursday 04 September 2008 17:51:56 Michael Bayer wrote:
On Sep 4, 2008, at 9:03 AM, [EMAIL PROTECTED] wrote:
i used to set sqlalchemy.util.Set to be
sqlalchemy.util.OrderedSet and that worked well...
if all those basic things (dict, set, odict, oset, etc) are
always routed via sqlachemy.util, then one can replace them with
whatever fancy. One main usage is that testing cases would be
more repeatable, because flush plans and other hash-relating
things will be same, if all sets are ordered and all dicts are
ordered.
this is not going to impact anything speedwise, it only means
changing several hundred places where {} or set() is used, and
keeping some discipline of not introducing those in future code.
someone may tell me about a way to directly hack pythons notion
of {} with something mine... would be good but is going to impact
speed of *any* code, not just SA.
mike, sorry for repeating myself again on the theme :-)
i can prepare The patch as long as u decide to keep such
protocol...
i believe we already have a layer in the test/ suite which can
globally replace imports with something specific, and it is being
used. It's Jason's thing but you can dig into the source to see
how it works.
nooo, u got me wrong. i'd like all the {} dict() set() usage
all-over-sa to be routed via util.dict and util.set (which default to
the builtins), so then one could easily replace them with whatever
s/he wants. i guess one could hack the python builtins/module or
globals() to replace the dict() and set() globally, but i dont think
{} is affected; still, such hack will affect any other code, and not
only SA.
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
dict
type 'dict'
globals()
{'__builtins__': module '__builtin__'
(built-in), '__name__': '__main__', '__doc__': None}
locals()
{'__builtins__': module '__builtin__'
(built-in), '__name__': '__main__', '__doc__': None}
__builtins__
module '__builtin__' (built-in)
__builtins__.dict
type 'dict'
class mydict( dict): pass
...
__builtins__.dict = mydict
dict
class '__main__.mydict'
type({})
type 'dict'
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
