Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 4:27 PM, Michael Bayer wrote: > as for the __slots__ thing, that's a separate issue.if your patch doesn't > break tests we can set that for 0.9 as well, I doubt anyone is subclassing > InstanceState, though I'd want to see what the speedup is with that. About 15% on s

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Michael Bayer
Here's a ticket where we can keep talking about this: http://www.sqlalchemy.org/trac/ticket/2720 (here's the py3k ticket also: http://www.sqlalchemy.org/trac/ticket/2161) note that SQLAlchemy 0.9 will no longer use 2to3, and will by Python 2.6->3.3 in place.The enhancement here is targeted

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 2:31 PM, Michael Bayer wrote: > did you generate your code here with pyrex? Oh, sorry, I didn't answer this. No. I wrote it by hand. Pyrex-generated code is inscrutable, not that there's any need to inscrute. But really, when using pyrex, the C file ought to be considere

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 2:20 PM, Claudio Freire wrote: > On Mon, May 6, 2013 at 1:50 PM, Michael Bayer > wrote: >> that's a lot of effort there. How confident are you that memory and >> references are handled correctly in the .c code? > > Quite. It's not my first C extension. But, truly, C is c

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 2:31 PM, Michael Bayer wrote: > did you generate your code here with pyrex?If you want to jump in and > rework our C extensions to be pyrex based and everything works out just as > well or better than before, it'll be a great 0.9/1.0 feature.I've got a > bit of ex

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Michael Bayer
did you generate your code here with pyrex?If you want to jump in and rework our C extensions to be pyrex based and everything works out just as well or better than before, it'll be a great 0.9/1.0 feature.I've got a bit of experience with cython already as I've worked on lxml a bit, cyt

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 1:50 PM, Michael Bayer wrote: > that's a lot of effort there. How confident are you that memory and > references are handled correctly in the .c code? Quite. It's not my first C extension. But, truly, C is complex. > That's a lot of C code, and it took years for us to ir

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Michael Bayer
that's a lot of effort there. How confident are you that memory and references are handled correctly in the .c code? That's a lot of C code, and it took years for us to iron out all the memory leaks in the existing C extensions that we had - the original author eventually stopped maintaining

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 1:10 PM, Claudio Freire wrote: > On Thu, May 2, 2013 at 3:34 PM, Claudio Freire wrote: >> Without the C extension: >>ncalls tottime percall cumtime percall filename:lineno(function) >> 20811734 27.8290.000 27.8550.000 attributes.py:171(__get__) >> 763

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Thu, May 2, 2013 at 3:34 PM, Claudio Freire wrote: > Without the C extension: >ncalls tottime percall cumtime percall filename:lineno(function) > 20811734 27.8290.000 27.8550.000 attributes.py:171(__get__) > 7631984 13.5320.000 31.8510.000 ruby.py:86(get_param

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-02 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:59 PM, Michael Bayer wrote: All attributes have to be expire-able and act as proxies for a database connection so I'm not really sure where to go with that.I'm not too thrilled about proposals to build in various "alternate performance" behaviors as the library s

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-30 Thread Claudio Freire
On Tue, Apr 30, 2013 at 8:06 PM, Michael Bayer wrote: >>try: >>from sqlalchemy.cinstrumented import InstrumentedGetter >>__get__ = InstrumentedGetter(globals()) >>__get__.__name__ = '__get__' >>del InstrumentedGetter >>except ImportError: >>def __get

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-30 Thread Michael Bayer
On Apr 30, 2013, at 6:26 PM, Claudio Freire wrote: > On Fri, Apr 26, 2013 at 9:09 PM, Claudio Freire > wrote: >> On Fri, Apr 26, 2013 at 9:01 PM, Michael Bayer >> wrote: >>> All attributes have to be expire-able and act as proxies for a database >>> connection so I'm not really sure

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-30 Thread Claudio Freire
On Fri, Apr 26, 2013 at 9:09 PM, Claudio Freire wrote: > On Fri, Apr 26, 2013 at 9:01 PM, Michael Bayer > wrote: >> All attributes have to be expire-able and act as proxies for a database >> connection so I'm not really sure where to go with that.I'm not too >> thrilled about p

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 9:01 PM, Michael Bayer wrote: > All attributes have to be expire-able and act as proxies for a database > connection so I'm not really sure where to go with that.I'm not too > thrilled about proposals to build in various "alternate performance" > beha

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Michael Bayer
On Apr 26, 2013, at 7:59 PM, Michael Bayer wrote: > > On Apr 26, 2013, at 7:56 PM, Claudio Freire wrote: > >> On Fri, Apr 26, 2013 at 8:47 PM, Claudio Freire >> wrote: >>> On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer >>> wrote: > Anyway, with that (fragile) change, I get a spe

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Michael Bayer
On Apr 26, 2013, at 7:56 PM, Claudio Freire wrote: > On Fri, Apr 26, 2013 at 8:47 PM, Claudio Freire > wrote: >> On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer >> wrote: >>> Anyway, with that (fragile) change, I get a speedup of 10% overall runtime, and about 50% alchemy-specific

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:47 PM, Claudio Freire wrote: > On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer > wrote: >> >>> Anyway, with that (fragile) change, I get a speedup of 10% overall >>> runtime, and about 50% alchemy-specific runtime. Considering I knew >>> about attribute access' slowness

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer wrote: > >> Anyway, with that (fragile) change, I get a speedup of 10% overall >> runtime, and about 50% alchemy-specific runtime. Considering I knew >> about attribute access' slowness and avoided it in my test, that has >> to account for something w

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Michael Bayer
On Apr 26, 2013, at 6:13 PM, Claudio Freire wrote: > Ultimately, though, it's InstrumentedAttribute.__get__ the one sucking > up 30% of alchemy-bound CPU time. I guess there's little that can be > done, since it's necessary to track state changes. But there's a neat > property of descriptors, wh

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 2:04 PM, Michael Bayer wrote: >>> are you referring to sqlalchemy/orm/loading.py ? I'd be pretty impressed >>> if you can find significant optimizations there which don't break usage >>> contracts.I've spent years poring over profiles and squeezing every >>> functi

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Michael Bayer
On Apr 26, 2013, at 12:41 PM, Claudio Freire wrote: > On Fri, Apr 26, 2013 at 1:35 PM, Michael Bayer > wrote: >>> >>> Um... a tad OT, but looking at that code, there's lots of >>> opportunities for optimization. >>> >>> I'll have to profile a bit and let you know. >> >> are you referring to

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 1:35 PM, Michael Bayer wrote: >> >> Um... a tad OT, but looking at that code, there's lots of >> opportunities for optimization. >> >> I'll have to profile a bit and let you know. > > are you referring to sqlalchemy/orm/loading.py ? I'd be pretty impressed if > you can f

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Michael Bayer
On Apr 26, 2013, at 12:24 PM, Claudio Freire wrote: > > Um... a tad OT, but looking at that code, there's lots of > opportunities for optimization. > > I'll have to profile a bit and let you know. are you referring to sqlalchemy/orm/loading.py ? I'd be pretty impressed if you can find sign

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 12:24 PM, Werner wrote: > On 26/04/2013 17:07, Claudio Freire wrote: >> >> On Fri, Apr 26, 2013 at 12:06 PM, Werner wrote: >>> >>> http://sqlalchemy.readthedocs.org/en/rel_0_8/orm/tutorial.html#querying >> >> >> Not entirely, if you don't use yield_per (as shown in the doc

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Werner
On 26/04/2013 17:07, Claudio Freire wrote: On Fri, Apr 26, 2013 at 12:06 PM, Werner wrote: On 26/04/2013 16:41, alonn wrote: so not to load too much into memory I should do something like: for i in session.query(someobject).filter(id>something) print i I'm guessing the answer is no, bec

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 12:06 PM, Werner wrote: > On 26/04/2013 16:41, alonn wrote: >> >> so not to load too much into memory I should do something like: >> >> for i in session.query(someobject).filter(id>something) >> print i >> >> I'm guessing the answer is no, because of the nature of sql,

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Werner
Hi, On 26/04/2013 16:41, alonn wrote: so not to load too much into memory I should do something like: for i in session.query(someobject).filter(id>something) print i I'm guessing the answer is no, because of the nature of sql, but I'm not an expert so I'm asking. yes you can, check out th

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Mauricio de Abreu Antunes
Query object has a __iter__ descriptor. 2013/4/26 Mauricio de Abreu Antunes > Read the source: > > def all(self): > """Return the results represented by this ``Query`` as a list. > > This results in an execution of the underlying query. > > """ > return list(self) > > it means t

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Mauricio de Abreu Antunes
Read the source: def all(self): """Return the results represented by this ``Query`` as a list. This results in an execution of the underlying query. """ return list(self) it means that this method collects everything it needs and it is yielded by the generator. If you returns th