Re: [sqlalchemy] how fast can pure sqlalchemy perform?

2015-08-10 Thread Claudio Freire
On Fri, Aug 7, 2015 at 6:58 PM, kk krm...@gmail.com wrote: On Friday 07 August 2015 10:05 PM, Claudio Freire wrote: On Fri, Aug 7, 2015 at 12:51 PM, Mike Bayer mike...@zzzcomputing.com wrote: On 8/7/15 11:05 AM, kk wrote: On Friday 07 August 2015 03:03 PM, Ladislav Lenart wrote: Hello

Re: [sqlalchemy] how fast can pure sqlalchemy perform?

2015-08-10 Thread Claudio Freire
On Mon, Aug 10, 2015 at 6:42 PM, kk krm...@gmail.com wrote: Hello, Tahnks to you and Mike for detaild insight, My questions follow in-line.On Friday 07 August 2015 08:48 PM, Claudio Freire wrote: On Fri, Aug 7, 2015 at 12:05 PM, kk krm...@gmail.com wrote: On Friday 07 August 2015 03:03 PM

Re: [sqlalchemy] how fast can pure sqlalchemy perform?

2015-08-07 Thread Claudio Freire
On Fri, Aug 7, 2015 at 12:05 PM, kk krm...@gmail.com wrote: On Friday 07 August 2015 03:03 PM, Ladislav Lenart wrote: Hello. ORM is certainly slower. How much depends A LOT on your workload. For example bulk operations with ORM are an order of magnitude slower than raw SQL. On the other

Re: [sqlalchemy] how fast can pure sqlalchemy perform?

2015-08-07 Thread Claudio Freire
On Fri, Aug 7, 2015 at 12:51 PM, Mike Bayer mike...@zzzcomputing.com wrote: On 8/7/15 11:05 AM, kk wrote: On Friday 07 August 2015 03:03 PM, Ladislav Lenart wrote: Hello. ORM is certainly slower. How much depends A LOT on your workload. For example bulk operations with ORM are an order

Re: [sqlalchemy] Handling big Python objects

2014-12-03 Thread Claudio Freire
On Wed, Dec 3, 2014 at 7:23 PM, Andrea Gavana andrea.gav...@gmail.com wrote: 3. As an aside, not zlib-ing the files saves about 5 seconds/simulation (over a 20 seconds save) but increases the database size by 4 times. I'll have to check if this is OK. You can usually specify a compression

Re: [sqlalchemy] objects not necessarily pulled from session's identity map when they should (?)

2014-11-04 Thread Claudio Freire
On Tue, Nov 4, 2014 at 3:15 PM, Jonathan Vanasco jvana...@gmail.com wrote: I have a permissions check routine that repeatedly queries for a certain Foo2Bar table class Foo2Bar(Base): __tablename__ = 'foo_2_bar' id_foo = Column(Integer, ForeignKey(foo.id),

Re: [sqlalchemy] statement.compile and legacy is not null code

2014-10-01 Thread Claudio Freire
On Wed, Oct 1, 2014 at 6:51 PM, Jonathan Vanasco jvana...@gmail.com wrote: # 0.7.9 and later sq = sess.query(A.id.label('id')).filter(A.id != None) I've used ~(id == None) since 0.3 quite successfully (though I've eventually switched to != None since it's way more natural so not sure if

Re: [sqlalchemy] Closing pool connections

2014-09-25 Thread Claudio Freire
On Thu, Sep 25, 2014 at 5:33 AM, Thierry Florac tflo...@gmail.com wrote: Hi, I have a multi-threaded web application using SQLAlchemy connections pool. As soon as a connection is opened, it's maintained opened in the pool even if it's not used anymore. On a long run, this can consume too much

Re: [sqlalchemy] Dangers of setting a session's expire_on_commit=False?

2014-07-25 Thread Claudio Freire
On Fri, Jul 25, 2014 at 7:55 PM, Michael Bayer mike...@zzzcomputing.com wrote: Don't objects potentially have this problem anyway, in the sense that if they are accessed TWICE after a commit, the second access will use the data cached from the first, and could again be out of date? only if

Re: [sqlalchemy] Dangers of setting a session's expire_on_commit=False?

2014-07-25 Thread Claudio Freire
On Fri, Jul 25, 2014 at 10:09 PM, Michael Bayer mike...@zzzcomputing.com wrote: For the record, I'm using expire_on_commit=off, because I also use model instances outside the scope of their originating transaction. I've had no problems with it, but I did have to be very careful with the

Re: [sqlalchemy] Mapping lots of similar tables / database design

2014-04-01 Thread Claudio Freire
On Tue, Apr 1, 2014 at 8:07 PM, Michael Bayer mike...@zzzcomputing.com wrote: Think about it. You have 2000 individual Python classes all called TimeSeries, all referring to a different table. This is the question you have to answer (and which I think is going to wind you up back at one

Re: [sqlalchemy] filter vs get question

2014-02-20 Thread Claudio Freire
On Thu, Feb 20, 2014 at 5:22 PM, Jonathan Vanasco jonat...@findmeon.com wrote: this seems to work, but I want to just make sure this is the intended behavior. a = dbSession.query( Something ).filter( Something.primary_key == 1 ).first() b = dbSession.query( Something ).get( 1 ) c =

Re: [sqlalchemy] Sharding without the ORM

2014-02-19 Thread Claudio Freire
On Wed, Feb 19, 2014 at 1:03 PM, Silviu Dicu silviud...@gmail.com wrote: Hi, I have a legacy application that doesn't use the ORM but I need to implement horizantal sharding on mysql. Did anybody has done something similar or have any ideas. You mean besides manually computing a consistent

[sqlalchemy] Cannot use pgbouncer's statement mode

2014-02-17 Thread Claudio Freire
Hello there. I'm using SQLAlchemy 0.7.10 with postgres+pgbouncer, and I'm trying to move from a transaction-mode pool towards a statement-mode pool. The reason for this move, is that I've got quite a high volume of reads that cannot be quickly serviced in transaction mode, since many connections

Re: [sqlalchemy] Cannot use pgbouncer's statement mode

2014-02-17 Thread Claudio Freire
On Mon, Feb 17, 2014 at 9:31 PM, Michael Bayer mike...@zzzcomputing.com wrote: the Python DBAPI as you know returns a connection that's in a transaction. So if you are using an environment that has zero tolerance for even two statements in one transaction, you should turn on autocommit at the

Re: [sqlalchemy] Prepared Statements in Postgresql

2014-02-12 Thread Claudio Freire
On Wed, Feb 12, 2014 at 5:03 PM, Tony Locke tlo...@tlocke.org.uk wrote: I've noticed some opinions online indicating that psycopg2 does not have prepared statement support (e.g. - http://www.cerias.purdue.edu/site/blog/post/beware_sql_injections_due_to_missing_prepared_statement_support/)

Re: [sqlalchemy] Session remove/close MySQL

2014-02-07 Thread Claudio Freire
I knew I should've been more explicit On Fri, Feb 7, 2014 at 1:07 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Feb 7, 2014, at 11:01 AM, Claudio Freire klaussfre...@gmail.com wrote: I've had similar issues with 0.7.10. SA opens an implicit transaction, incorrect, DBAPI does

Re: [sqlalchemy] Session remove/close MySQL

2014-02-07 Thread Claudio Freire
On Fri, Feb 7, 2014 at 2:35 PM, Michael Bayer mike...@zzzcomputing.com wrote: The connection pool, if in use, will then not actually close the connection if it is to remained pooled, it calls rollback() as part of the pool release mechanism. Recent versions of SQLAlchemy allow this to show up

Re: [sqlalchemy] Session remove/close MySQL

2014-02-07 Thread Claudio Freire
On Fri, Feb 7, 2014 at 11:51 AM, Simon King si...@simonking.org.uk wrote: On Fri, Feb 7, 2014 at 9:28 AM, Christian Démolis christiandemo...@gmail.com wrote: Hi all, Actually, i have some problem closing my session... I tried using scopedsession with session.remove I tried using normal

Re: [sqlalchemy] prepared statements

2014-02-06 Thread Claudio Freire
On Thu, Feb 6, 2014 at 3:08 PM, Michael Bayer mike...@zzzcomputing.com wrote: I was imagining/hoping I'd find something like this: # prepare the query: myPreparedQuery = mySession.query(stuff).filter(parameter definitions).prepare() # run the query whenever I need it during my

Re: [sqlalchemy] prepared statements

2014-02-06 Thread Claudio Freire
On Thu, Feb 6, 2014 at 4:00 PM, Jonathan Vanasco jonat...@findmeon.com wrote: just wondering -- would it be possible to mimic this behavior using a custom view for this select ( in postgresql ) and then querying that ? i think the query planner might only run on the creation. I think views

Re: [sqlalchemy] prepared statements

2014-02-06 Thread Claudio Freire
On Thu, Feb 6, 2014 at 4:29 PM, Rick Otten rottenwindf...@gmail.com wrote: Thanks Claudio - I'll mull over the pros and cons of explicitly managing the connections to prepare the statements vs just being patient while the job runs vs gains I might achieve elsewhere. Remember, prepared

Re: [sqlalchemy] prepared statements

2014-02-06 Thread Claudio Freire
On Thu, Feb 6, 2014 at 4:38 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Feb 6, 2014, at 1:31 PM, Claudio Freire klaussfre...@gmail.com wrote: There are various performance concerns related to SELECT queries and solving them depends much on identifying where a particular performance

Re: [sqlalchemy] prepared statements

2014-02-06 Thread Claudio Freire
On Thu, Feb 6, 2014 at 4:47 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Feb 6, 2014, at 2:43 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, Feb 6, 2014 at 4:38 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Feb 6, 2014, at 1:31 PM, Claudio Freire klaussfre

Re: [sqlalchemy] Strange behaviour in multi-process environment

2014-01-20 Thread Claudio Freire
On Mon, Jan 20, 2014 at 1:51 PM, Richard Gerd Kuesters rich...@humantech.com.br wrote: i was having this kind of problem while using a multi-threaded app, but with a postgres backend. in postgres, with high-concurrency, i was expecting this kind of behaviour, so i had to implement some simple

Re: [sqlalchemy] Why ORM objects in a session expire immediately after a failure in flush?

2014-01-10 Thread Claudio Freire
On Thu, Jan 9, 2014 at 9:59 PM, Michael Bayer mike...@zzzcomputing.com wrote: I do notice that you’re catching an IntegrityError.The typical pattern when writing code that wants to catch these and then continue is to run the individual set of questionable operations within a SAVEPOINT, that

Re: [sqlalchemy] Custom logic for query execution

2014-01-10 Thread Claudio Freire
On Fri, Jan 10, 2014 at 3:50 PM, Sylvester Steele sylvesterste...@gmail.com wrote: Hi, Currently I am using the sqlalchemy engine to execute string queries only. I do plan on using sqlalchemy more extensively (including ORM) in the near future. I need to add retry logic on every query, in

Re: [sqlalchemy] SQLAlchemy 0.9.1 released

2014-01-08 Thread Claudio Freire
Typo: when condition is non-empty should be when conditionS is non-empty On Wed, Jan 8, 2014 at 1:53 PM, Michael Bayer mike...@zzzcomputing.com wrote: a new section has been added as the first “Core Behavioral Change”:

Re: [sqlalchemy] Speed up bulk inserts

2013-11-13 Thread Claudio Freire
On Wed, Nov 13, 2013 at 2:04 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Nov 13, 2013, at 11:52 AM, Claudio Freire klaussfre...@gmail.com wrote: On Wed, Nov 13, 2013 at 1:45 PM, Michael Bayer mike...@zzzcomputing.com wrote: Am Mittwoch, 6. November 2013 21:58:53 UTC+1 schrieb

Re: [sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?

2013-10-07 Thread Claudio Freire
On Mon, Oct 7, 2013 at 12:01 PM, Jonathan Vanasco jonat...@findmeon.com wrote: so the popular workaround(s) are to do both: 1- defer to thread (which most people would do anyways) 2- open/close multiple sessions (which are cheap). basically , instead of approaching a task in twisted as a

Re: [sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?

2013-10-03 Thread Claudio Freire
On Thu, Oct 3, 2013 at 6:02 PM, Michael Bayer mike...@zzzcomputing.com wrote: well there's always gevent. It might even be possible to get twisted and gevent to work together. So far I've managed to stick with multiprocessing for any kind of parallelism and it's done fine, but I'm not

Re: [sqlalchemy] What is declarative_base() exactly?

2013-09-23 Thread Claudio Freire
On Mon, Sep 23, 2013 at 10:22 AM, Edward Kim onward.ed...@gmail.com wrote: Base = declarative_base() Base class 'sqlalchemy.ext.declarative.api.Base' How this function is return class, not instance? Is it kind of design pattern? I know It is not a big deal for just using SQLAlchemy, but

Re: [sqlalchemy] Why no Raw Query Builder?

2013-08-10 Thread Claudio Freire
On Sat, Aug 10, 2013 at 11:31 AM, Taba Taba betak...@gmail.com wrote: Thank you so much. Another question: $this-select(col1); if(1 0) { $this-select(col2, col3); } $this-from(tbl)-where(1 = 1); if( 2 1) { $this-where(2 1); } $this-left_outer_join(tbl2, tbl2.t_id = tbl.id);

Re: [sqlalchemy] sorry, too many clients already

2013-07-26 Thread Claudio Freire
On Thu, Jul 25, 2013 at 8:39 PM, kris kkvilek...@gmail.com wrote: On Thursday, July 25, 2013 4:12:50 PM UTC-7, Klauss wrote: On Thu, Jul 25, 2013 at 7:58 PM, kris kkvil...@gmail.com wrote: My postgres.conf has a parameter max_connections = 100 That's not only the default, but it's also

Re: [sqlalchemy] sorry, too many clients already

2013-07-25 Thread Claudio Freire
On Thu, Jul 25, 2013 at 7:32 PM, kris kkvilek...@gmail.com wrote: We are running a replicated (multiple machines behind ngnix) Turbogears 2.1.5 App With SA 0.7.8 and postgres 8.4 After a lot of recent usage the system ground to halt and we are receiving (OperationalError) FATAL: sorry, too

Re: [sqlalchemy] sorry, too many clients already

2013-07-25 Thread Claudio Freire
On Thu, Jul 25, 2013 at 7:58 PM, kris kkvilek...@gmail.com wrote: My postgres.conf has a parameter max_connections = 100 That's not only the default, but it's also not really recommended to push it much higher, so only do so if you really need a big pool on each machine, and if you're sure

Re: [sqlalchemy] Query caching allows two objects representing same underlying data in session?

2013-07-16 Thread Claudio Freire
On Mon, Jul 15, 2013 at 8:02 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Jul 15, 2013, at 4:51 PM, Amir Elaguizy aelag...@gmail.com wrote: I'm having this weird problem using the query caching recipes in which two instances of a model representing the same underlying dataset will

Re: [sqlalchemy] Query caching allows two objects representing same underlying data in session?

2013-07-16 Thread Claudio Freire
On Tue, Jul 16, 2013 at 4:22 PM, Michael Bayer mike...@zzzcomputing.com wrote: To put a cached instance into a session, you must first copy it, then update. How to do that, is very application-specific, and I don't think it can be automated. that's what merge(don't_load=True) does. If

Re: [sqlalchemy] Query and compiled_cache

2013-06-07 Thread Claudio Freire
On Thu, Jun 6, 2013 at 2:20 PM, Michael Bayer mike...@zzzcomputing.com wrote: diff --git a/lib/sqlalchemy/orm/query.py b/lib/sqlalchemy/orm/query.py index c2ec72c..b458975 100644 --- a/lib/sqlalchemy/orm/query.py +++ b/lib/sqlalchemy/orm/query.py @@ -40,12 +40,15 @@ from ..sql import (

Re: [sqlalchemy] Query and compiled_cache

2013-06-07 Thread Claudio Freire
On Fri, Jun 7, 2013 at 1:56 PM, Claudio Freire klaussfre...@gmail.com wrote: This is my current BakedQuery: class BakedQuery(sqlalchemy.orm.query.Query): F**k gmail again. Why the F+!k doesn't it show me that it'll bork all whitespace before I hit send... I smell a bug report coming

Re: [sqlalchemy] Query and compiled_cache

2013-06-06 Thread Claudio Freire
On Tue, Jun 4, 2013 at 5:26 PM, Claudio Freire klaussfre...@gmail.com wrote: On Sun, Jun 2, 2013 at 9:41 PM, Claudio Freire klaussfre...@gmail.com wrote: So the whole thing is rolled up into the named thing I referred to also, so that there's no need to keep a Query object hanging around

Re: [sqlalchemy] Query and compiled_cache

2013-06-04 Thread Claudio Freire
On Sun, Jun 2, 2013 at 9:41 PM, Claudio Freire klaussfre...@gmail.comwrote: So the whole thing is rolled up into the named thing I referred to also, so that there's no need to keep a Query object hanging around, when we say bake() we're really just referring to a position in the code somewhere

Re: [sqlalchemy] Query and compiled_cache

2013-06-02 Thread Claudio Freire
On Sat, Jun 1, 2013 at 10:59 PM, Michael Bayer mike...@zzzcomputing.comwrote: The recipe on the wiki also has the issue that it isn't even caching anything to do with the QueryContext, including all of this information regarding eager joins which is pretty important. Your modifications try to

Re: [sqlalchemy] Query and compiled_cache

2013-06-02 Thread Claudio Freire
On Mon, Jun 3, 2013 at 1:55 AM, Michael Bayer mike...@zzzcomputing.comwrote: Well, if that works, it certainly covers my needs so there would be no pressing need to incorporate it into the core. I'll let you know tomorrow. It's so straightforward at this point I'm leaning towards some more

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Thu, May 30, 2013 at 7:04 PM, Michael Bayer mike...@zzzcomputing.com wrote: The hashing thing really has to start as a core concept first. It's a big job but would be very helpful for caching scenarios and would allow us to build this feature on Query without too much difficulty. The

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Fri, May 31, 2013 at 11:29 AM, Michael Bayer mike...@zzzcomputing.com wrote: On Friday, May 31, 2013 10:18:41 AM UTC-4, Klauss wrote: On Thu, May 30, 2013 at 7:04 PM, Michael Bayer mik...@zzzcomputing.com wrote: The hashing thing really has to start as a core concept first. It's a

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Fri, May 31, 2013 at 12:46 PM, Michael Bayer mike...@zzzcomputing.com wrote: this hash works for backends that can render LIMIT as a bound parameter. It will *not* work for SQL server which cannot render LIMIT as a bound parameter. If the hash is determined at the level of Query, we *do

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Thu, May 30, 2013 at 3:28 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, May 30, 2013 at 2:25 PM, Michael Bayer mike...@zzzcomputing.com wrote: On May 30, 2013, at 11:25 AM, Claudio Freire klaussfre...@gmail.com wrote: That way, one could use the second form up there and benefit

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Fri, May 31, 2013 at 4:03 PM, Claudio Freire klaussfre...@gmail.comwrote: With: class CacheableQuery(sqlalchemy.orm.query.Query): def __init__(self, *p, **kw): self._cached_context = None self._cached_context_labels = None super(CacheableQuery, self).__init__(*p, **kw

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Fri, May 31, 2013 at 4:44 PM, Michael Bayer mike...@zzzcomputing.comwrote: can you just attach a working .py script How does that work without a database? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and

Re: [sqlalchemy] Query and compiled_cache

2013-05-31 Thread Claudio Freire
On Fri, May 31, 2013 at 4:47 PM, Claudio Freire klaussfre...@gmail.comwrote: On Fri, May 31, 2013 at 4:44 PM, Michael Bayer mike...@zzzcomputing.comwrote: can you just attach a working .py script How does that work without a database? Ok, I took one of SQLA's tests, and make it break

[sqlalchemy] Query and compiled_cache

2013-05-30 Thread Claudio Freire
I know this has been discussed a great deal already, but I've been noticing this: SomeClass.query() .filter_by(id = blah) .execution_options(compiled_cache = _orm_query_cache) .options( orm.joinedload(blah), orm.joinedload(blah, bleh), orm.joinedload(blah,

Re: [sqlalchemy] Query and compiled_cache

2013-05-30 Thread Claudio Freire
On Thu, May 30, 2013 at 2:25 PM, Michael Bayer mike...@zzzcomputing.com wrote: On May 30, 2013, at 11:25 AM, Claudio Freire klaussfre...@gmail.com wrote: That way, one could use the second form up there and benefit from query hashing, because session/param binding wouldn't change the hash

Re: [sqlalchemy] Query and compiled_cache

2013-05-30 Thread Claudio Freire
On Thu, May 30, 2013 at 3:45 PM, Michael Bayer mike...@zzzcomputing.com wrote: On May 30, 2013, at 2:28 PM, Claudio Freire klaussfre...@gmail.com wrote: 4. it would have a super crapload of very complete and clean unit tests. Ehm... I would imagine all the current tests involving Query

Re: [sqlalchemy] Query and compiled_cache

2013-05-30 Thread Claudio Freire
On Thu, May 30, 2013 at 4:19 PM, Michael Bayer mike...@zzzcomputing.com wrote: In case I'm not clear, this would not be cached if I were to take id(internals) query(Blah).filter(blah).join(blah).first() But I don't care, because that's expensive on its own. result caching (because we're

Re: [sqlalchemy] Can SQLAlchemy execute multiple select statements in a single round trip?

2013-05-23 Thread Claudio Freire
On Thu, May 23, 2013 at 3:32 PM, Michael Bayer mike...@zzzcomputing.com wrote: If not, it would be nice to call .future() or .promise() on instead of a ` sqlalchemy.orm.query.Query` instance instead of .all() to batch multiple queries and have them executed in a single round trip. The way

Re: [sqlalchemy] Can SQLAlchemy execute multiple select statements in a single round trip?

2013-05-23 Thread Claudio Freire
On Thu, May 23, 2013 at 4:21 PM, Michael Bayer mike...@zzzcomputing.com wrote: In postgres, it could be implemented with Async I/O and multiple cursors, but sadly Async is something of a global pool configuration, not something you can turn on/off per call. IMHO stuffing async calls and such

Re: [sqlalchemy] EAV Optimizations

2013-05-16 Thread Claudio Freire
On Thu, May 16, 2013 at 7:16 PM, Simon King si...@simonking.org.uk wrote: I don't think this would be a good match for HSTORE - I assume that retrieving values from HSTORE won't be as efficient as retrieving them from their own rows. It depends on the number of attributes per test. It won't

Re: [sqlalchemy] EAV Optimizations

2013-05-14 Thread Claudio Freire
On Tue, May 14, 2013 at 12:38 PM, Michael Bayer mike...@zzzcomputing.com wrote: storing pickles in the database is an awful idea since your data is now specific not just to Python but to the current shape of your object model at a point in time. Storing JSON is not acceptable for the case

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Thu, May 2, 2013 at 3:34 PM, Claudio Freire klaussfre...@gmail.com wrote: Without the C extension: ncalls tottime percall cumtime percall filename:lineno(function) 20811734 27.8290.000 27.8550.000 attributes.py:171(__get__) 7631984 13.5320.000 31.8510.000

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 1:10 PM, Claudio Freire klaussfre...@gmail.com wrote: On Thu, May 2, 2013 at 3:34 PM, Claudio Freire klaussfre...@gmail.com wrote: Without the C extension: ncalls tottime percall cumtime percall filename:lineno(function) 20811734 27.8290.000 27.855

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 1:50 PM, Michael Bayer mike...@zzzcomputing.com wrote: that's a lot of effort there. How confident are you that memory and references are handled correctly in the .c code? Quite. It's not my first C extension. But, truly, C is complex. That's a lot of C code, and it

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 2:31 PM, Michael Bayer mike...@zzzcomputing.com wrote: did you generate your code here with pyrex?If you want to jump in and rework our C extensions to be pyrex based and everything works out just as well or better than before, it'll be a great 0.9/1.0 feature.

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 2:31 PM, Michael Bayer mike...@zzzcomputing.com wrote: did you generate your code here with pyrex? Oh, sorry, I didn't answer this. No. I wrote it by hand. Pyrex-generated code is inscrutable, not that there's any need to inscrute. But really, when using pyrex, the C

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-06 Thread Claudio Freire
On Mon, May 6, 2013 at 4:27 PM, Michael Bayer mike...@zzzcomputing.com wrote: as for the __slots__ thing, that's a separate issue.if your patch doesn't break tests we can set that for 0.9 as well, I doubt anyone is subclassing InstanceState, though I'd want to see what the speedup is with

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-05-02 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:59 PM, Michael Bayer mike...@zzzcomputing.com wrote: All attributes have to be expire-able and act as proxies for a database connection so I'm not really sure where to go with that.I'm not too thrilled about proposals to build in various alternate performance

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-30 Thread Claudio Freire
On Fri, Apr 26, 2013 at 9:09 PM, Claudio Freire klaussfre...@gmail.com wrote: On Fri, Apr 26, 2013 at 9:01 PM, Michael Bayer mike...@zzzcomputing.com wrote: All attributes have to be expire-able and act as proxies for a database connection so I'm not really sure where to go with that.I'm

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-30 Thread Claudio Freire
...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out. /* instrumented.c Copyright (C) 2013 Claudio Freire klaussfre...@gmail.com

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 12:06 PM, Werner werner.bru...@sfr.fr wrote: On 26/04/2013 16:41, alonn wrote: so not to load too much into memory I should do something like: for i in session.query(someobject).filter(idsomething) print i I'm guessing the answer is no, because of the nature of

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 12:24 PM, Werner werner.bru...@sfr.fr wrote: On 26/04/2013 17:07, Claudio Freire wrote: On Fri, Apr 26, 2013 at 12:06 PM, Werner werner.bru...@sfr.fr wrote: http://sqlalchemy.readthedocs.org/en/rel_0_8/orm/tutorial.html#querying Not entirely, if you don't use

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 1:35 PM, Michael Bayer mike...@zzzcomputing.com wrote: Um... a tad OT, but looking at that code, there's lots of opportunities for optimization. I'll have to profile a bit and let you know. are you referring to sqlalchemy/orm/loading.py ? I'd be pretty impressed if

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 2:04 PM, Michael Bayer mike...@zzzcomputing.com wrote: are you referring to sqlalchemy/orm/loading.py ? I'd be pretty impressed if you can find significant optimizations there which don't break usage contracts.I've spent years poring over profiles and squeezing

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer mike...@zzzcomputing.com wrote: Anyway, with that (fragile) change, I get a speedup of 10% overall runtime, and about 50% alchemy-specific runtime. Considering I knew about attribute access' slowness and avoided it in my test, that has to account

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 8:47 PM, Claudio Freire klaussfre...@gmail.com wrote: On Fri, Apr 26, 2013 at 8:15 PM, Michael Bayer mike...@zzzcomputing.com wrote: Anyway, with that (fragile) change, I get a speedup of 10% overall runtime, and about 50% alchemy-specific runtime. Considering I knew

Re: [sqlalchemy] Are sqlalchemy queries a generator?

2013-04-26 Thread Claudio Freire
On Fri, Apr 26, 2013 at 9:01 PM, Michael Bayer mike...@zzzcomputing.com wrote: All attributes have to be expire-able and act as proxies for a database connection so I'm not really sure where to go with that.I'm not too thrilled about proposals to build in various alternate performance

Re: [sqlalchemy] Doing reflection with multiple remote databases.

2013-01-16 Thread Claudio Freire
On Tue, Jan 15, 2013 at 8:48 PM, Hetii ghet...@gmail.com wrote: Even when i dump all of them into declarative base model, its still huge amount of data that need to be parsed and loaded. I want to ask if its possible to share table/column definition across different database models to reduce

Re: [sqlalchemy] [Q] Lack of isolation in unit tests

2012-09-18 Thread Claudio Freire
On Tue, Sep 18, 2012 at 11:07 AM, Ladislav Lenart lenart...@volny.cz wrote: def create_base(): return declarative_base(cls=_Base) Move the declaration of _Base to within create_base, and I think that should fix your problem. (I've had a similar one, not with test cases, but with a replica

Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 12:50 AM, Michael Bayer mike...@zzzcomputing.com wrote: Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. flush periodically, and don't maintain references to things you're done with.  

Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...

2012-05-30 Thread Claudio Freire
On Wed, May 30, 2012 at 8:39 PM, Michael Bayer mike...@zzzcomputing.com wrote: well yes, the way you're doing this is entirely the opposite of how the ORM is designed to function.    The Session has been developed in order to work in an intelligent manner with full graphs of interrelated

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Claudio Freire
On Thu, May 17, 2012 at 10:21 AM, Michael Bayer mike...@zzzcomputing.com wrote: There's a few different parts to what you're asking. The first is that you're comparing Python's use of OS memory (I'm assuming this is the 200+ MB) to Python's actual amount of objects present.   This is a common

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Claudio Freire
On Thu, May 17, 2012 at 11:57 AM, Vlad K. v...@haronmedia.com wrote: PostgreSQL 9.0.7, libpq is part of the same version -devel package PostgreSQL 9.1.3 (via pgsql yum repo), libpq is part of the same version -devel package 9.1 (the one you have in production) also has leak-related fixes. They

Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects

2012-05-09 Thread Claudio Freire
On Wed, May 9, 2012 at 4:11 PM, Michael Bayer mike...@zzzcomputing.com wrote: Hello, is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes.     you'd want

Re: [sqlalchemy] QueuePool limit size reached, using expression API only

2012-04-09 Thread Claudio Freire
On Mon, Apr 9, 2012 at 4:03 PM, Michael Bayer mike...@zzzcomputing.com wrote: close your connections after you are finished with them. They should be automatically returned to the pool when unreferenced. The OP may be storing stray references somewhere, or associating them somehow to a

Re: [sqlalchemy] QueuePool limit size reached, using expression API only

2012-04-09 Thread Claudio Freire
On Mon, Apr 9, 2012 at 4:36 PM, Michael Bayer mike...@zzzcomputing.com wrote: Using context managers, i.e. with engine.connect() as conn, is the most straightforward. IIRC, context managers are new in SA, aren't they? -- You received this message because you are subscribed to the Google

Re: [sqlalchemy] QueuePool limit size reached, using expression API only

2012-04-09 Thread Claudio Freire
On Mon, Apr 9, 2012 at 6:50 PM, Michael Bayer mike...@zzzcomputing.com wrote: better one to use from engine is begin() (also new in 0.7.6): with engine.begin() as conn:    ... that way everything you do with conn is on the same transaction. Yeah, because I'm using 0.5.8 (and couldn't switch

Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities

2012-03-05 Thread Claudio Freire
On Sun, Feb 19, 2012 at 6:27 AM, Andrey Popp 8may...@gmail.com wrote: I've managed quite efficient inlining with bytecode magic. It's especially effective with SQLA code, since it also specializes the inlined function, removing a lot of dead code (in the call context). That's pretty

Re: [sqlalchemy] postgres with .. select queries

2012-03-01 Thread Claudio Freire
On Thu, Mar 1, 2012 at 12:11 PM, A.M. age...@themactionfaction.com wrote: Well, the SQL standard points at WITH RECURSIVE which is more general anyway. W.R. is basically an inductive query loop construct (base case UNION induction step) where CONNECT BY only handles key-based tree retrieval,

Re: [sqlalchemy] postgres with .. select queries

2012-03-01 Thread Claudio Freire
On Thu, Mar 1, 2012 at 5:08 PM, Michael Bayer mike...@zzzcomputing.com wrote: OK also, this is ready to go in from my perspective, I don't make usage of CTEs in my normal work at the moment, so hopefully those people here interested in the feature can give this a review, maybe even try the

Re: [sqlalchemy] Sequences, Primary Keys, Relations

2012-02-23 Thread Claudio Freire
On Thu, Feb 23, 2012 at 4:49 PM, Adam Tauno Williams awill...@whitemice.org wrote: This works fine.  But if I create a Project object I can't relate it to a ProjectInfo object within the same transaction without calling flush() first.  Is there some way to encourage SQLalchemy to allocate a

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-22 Thread Claudio Freire
On Wed, Feb 22, 2012 at 4:29 PM, Michael Bayer mike...@zzzcomputing.com wrote: thanks for your reply. I haven't yet tested this with a profiler to see exactly what exactly is happening, but the bottom line is that the overall memory use grows with each iteration (or transaction processed), to

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-22 Thread Claudio Freire
On Wed, Feb 22, 2012 at 5:40 PM, Michael Bayer mike...@zzzcomputing.com wrote: Saw that a bit, but looking at the tips at the bottom, concrete implementation changes are not coming to mind.   An eternal structure is ubiquitous in any programming language.  sys.modules is a big list of all the

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-22 Thread Claudio Freire
On Wed, Feb 22, 2012 at 5:51 PM, Claudio Freire klaussfre...@gmail.com wrote: Such caches, for instance, are better made limited in lifespan (say, giving them a finite lifetime, making them expire, actively cleaning them from time to time). Structures that are truly required to be eternal

Re: [sqlalchemy] Re: Working with large IN lists

2012-02-22 Thread Claudio Freire
On Wed, Feb 22, 2012 at 6:21 PM, Michael Bayer mike...@zzzcomputing.com wrote: IMHO the whole point of using a high level, interpreted language like Python is that we don't have to be bogged down thinking like C programmers.   How come I've never had a memory fragmentation issue before ?      

Re: [sqlalchemy] DetachedInstanceError after transaction

2012-02-14 Thread Claudio Freire
On Tue, Feb 14, 2012 at 6:48 PM, Jonathan Vanasco jonat...@findmeon.com wrote: I will admit that i have some janky db sessioning stuff going on behind the scenes as I get used to pyramid and the new sqlalchemy. Then I'd say the janky db sessioning stuff going on behind the scenes is closing

Re: [sqlalchemy] Re: DetachedInstanceError after transaction

2012-02-14 Thread Claudio Freire
On Tue, Feb 14, 2012 at 7:19 PM, Jonathan Vanasco jonat...@findmeon.com wrote: could anyone point in the right direction to either: 1. rebind an object to a new session or 2. allow objects to still be 'read' in the detached state ? Eric said it best: replace commit with DBSession.flush() --

[sqlalchemy] PDF documentation 0.5

2012-02-09 Thread Claudio Freire
Hi there. I just tried to download the PDF documentation for 0.5 from the documentation page[0], and the link seems to be broken (404). Sorry if it has been brought to the list already, first time post here, I checked the archive and couldn't find anything related. [0]

Re: [sqlalchemy] PDF documentation 0.5

2012-02-09 Thread Claudio Freire
On Thu, Feb 9, 2012 at 1:28 PM, Michael Bayer mike...@zzzcomputing.com wrote: I've copied all the existing PDFs to my host here: http://www.sqlalchemy.org/doc_pdfs/ you can yank it from there. Got it. Thanks :-) -- You received this message because you are subscribed to the Google Groups