Re: Gratisan {?- Watch Godzilla (2014) Full Free Movie OnLine
*DANGER!!! do not want to click on the link above, yesterday I already click the link above and my computer a virus,just want to watch the movie but they actually gave me a false information, may god forgive their sins,but after I searched on google I finally found a link that right to watch this movie,please click herehttp://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you. http://tubemovies.biz/godzilla-2014-full-movieI've downloaded and watched it last night,I have searched google, I have found a place with an EASY PROCESS,UNBUFFERED when watching Online, and UNLIMITED SPEEDS when downloading,I recomended you for visit this place,http://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you..Do not believe the link above, he just gave misleading information,I've been watching this movie for free herehttp://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you. http://tubemovies.biz/godzilla-2014-full-moviehey man!!! Why do you deceive them by giving false information? do not believe this asshole,I've watched this movie for free athttp://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you..you're such a pig!!! Why are you giving misleading information to us?I've watched this movie for free even I can download it,please you download view the movie herehttp://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you. http://tubemovies.biz/godzilla-2014-full-movieeven I had to download it there.I am sure you and your family will love this movie,because this movie is the best movie I've ever watched,congratulations watch this movie is very goodhttp://tubemovies.biz/godzilla-2014-full-movie http://tubemovies.biz/godzilla-2014-full-movieorhttp://bestmovies.nation2.com/index.php?page=1141565422 http://bestmovies.nation2.com/index.php?page=1141565422JUST 2 STEP1. Click Button PLAY/WATCH NOW2. Create your Account/Sign in (if you a member)Done, You just choose the movie, you want to watch / downloadHopefully my experience can help you.,..* -- You received this message because you are subscribed to the Google Groups sqlalchemy-alembic group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-alembic+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Re: Postgresql 9.4 JSONB support
On Tuesday, June 24, 2014 6:21:40 PM UTC+4, Jonathan Vanasco wrote: On Tuesday, June 24, 2014 3:32:41 AM UTC-4, Damian Dimmich wrote: I intend to develop further syntax/query support for this type on an as needed basis for now - suggestions and comments are much appreciated. 1. Nice work! Now I'm more likely to use jsonb! :) 2. I ran into some sqlalchemy issues with postgresql and gin indexes recently. the default `.match()` didn't accept the regconfig argument, which can be a necessary argument to properly hint the query-planner to use the index. the result was the index is ignored and the column was being cast into ts_vector in real-time. on a few thousand rows, there was a 100x difference in query time.( discussion here : https://bitbucket.org/zzzeek/sqlalchemy/issue/3078/postgresql-full-text-search-on-match ; there's a PR in the queue too ) so my suggestion would be to make sure you run the results of the generated SQL through `EXPLAIN ANALYZE` on a decently large test dataset to make sure the gin indexes are being used properly. You could end up the situation were the sql is totally valid and looks right, but it's structured in a way that the indexes are ignored. You are way ahead of me in this case it seems :). I'll make sure to do that however - good suggestion. It looks like the final version of 9.4 will have more operator support than i'd originally expected (or it may be pushed into 9.5) according to my understanding of some of the commits in the postgres tree. It would be cool to have something working at around the 9.4 release date. d -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] inheriting from mapped classes
I read it, but could you illustrate it with a sample code based on the classic User/Addresses example? On Wednesday, June 25, 2014 11:45:12 AM UTC-4, Jonathan Vanasco wrote: On Tuesday, June 24, 2014 9:40:02 PM UTC-4, Victor Olex wrote: What I aiming for is to provide users a library of base class(es), which are mapped using SQLAlchemy. The classes are then meant to be extended by users with business logic. I am not hell bent on using inheritance for this, but for now I went with your __abstract__ = True solution only in a somewhat inverted way. There's another thread from within the past 2 weeks from someone else trying to tackle this problem. I mentioned in that post, and I'll mention again here -- the best method I found was to use a form of a registry pattern -- where child classes inherit from base class and mention any overrides + the types of relationships they require or provide. As the classes are initialized, this data is recorded in a registry. after initialization, relationships are mapped onto the classes uses the data in the registry. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] inheriting from mapped classes
On Thursday, June 26, 2014 10:38:50 AM UTC-4, Victor Olex wrote:
I read it, but could you illustrate it with a sample code based on the
classic User/Addresses example?
This is some pseudocode just to get the point across.
I use 2 files, one for the base class, the other for the user class.
The base class defines the objects, the other class extends them
In practice, I build up the registry dynamically by inspecting all the
items in the user library to see if they inherit from `_CoreObject`, then
figuring out which object they implement
##
# your core library
##
class _CoreObject(object):
pass
class _User(_CoreObject):
id = Column(Integer, Sequence('user_id_seq'), primary_key=True)
name = Column(String(50))
fullname = Column(String(50))
password = Column(String(12))
class _Address(_CoreObject):
id = Column(Integer, primary_key=True)
email_address = Column(String, nullable=False)
user_id = Column(Integer, ForeignKey('users.id'))
def setup_relationships( registry ):
registry['Address'].user = relationship(registry['User'],
backref=backref('addresses', order_by=registry['Address'].id))
##
# user library
##
class User(_User, Base):
__tablename__ = 'users'
class Address(_Address, Base):
__tablename__ = 'addresses'
registry = {
'User' : User,
'Address' : Address,
}
setup_relationships(registry)
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Performance problem of simple orm usage
On 6/26/14, 11:49 AM, Hans-Peter Jansen wrote: Dear SQLAchemistas, I'm suffering from a performance problem from a simple sequence like this: rec is a sensor record coming redundantly from the network, enos is a sensor device, ev is a sensor value record: def store_enos_rec(self, rec, srelay): enos = self.lookup_enos(rec.id, srelay) ts = rec.ts.replace(tzinfo = None) ev = self.session.query(EnOSValues).filter(EnOSValues.enos_id == enos.id, EnOSValues.ts == ts).first() if ev: return # create record ev = EnOSValues(enos_id = enos.id, ts = ts, status = True, temp = rec.temp) enos.values.append(ev) try: self.session.commit() except IntegrityError: self.session.rollback() This is running as a multiprocessing.Process on its own, but the self.session.commit call is the bottleneck here, as shown in the profile callgraph. Is there something simple, that can be done to mitigate the issue, other than doing it straight down the sql way? Obviously, it is taking a lot of time in history mgmt. A former approach to simply dump the record without searching duplicates and catching the IntegrityError took even more time due to the session rollback. Thanks for your patience and any enlightenment, It's very difficult to read this SVG and a traditional Python profile output sorted by calltime would be helpful, but what I can see is that you're calling commit() 20K times, so that is a lot. commit() is not intended to be something you call for every individual row of a large dataset, it is better that you feed a few thousand objects into the Session at a time before flushing and committing. I'm not sure what kind of application this is but I would not be mixing transactional control, that is the commit/rollback, inside of a business method that only seeks to create some new objects. There'd be a containing pattern within which store_enos_rec() is called these 20K times, then the commit() is called on the outside of that, or at least every 1000 records or so. Also, the create if not present pattern is often much better set up by preloading all the objects you'll deal with up front: existing_evs = dict(Session.query(EnosValues.enos_id, EnosValues)) if ev_id not in existing_evs: Session.add(EnosValues(...)) If you truly need to optimize a method whose job it is to start a new transaction, insert a single row if not present and then commit the transaction fully, I'd use a core insert() construct for that. Spinning up a whole unit of work in that context is overkill if performance is a concern. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered
I'm using SQLAlchemy with web.py, and have used it many times in the past. I'm working on a project using gevent/greenlets, and everything has been fine for a couple of months, until today. Suddenly, I'm getting these at a time when I'm not even receiving any requests. Does anyone have any advice or steps to troubleshoot? Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in
Re: [sqlalchemy] AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered
that looks pretty much like modules are being reloaded in process. Either get web.py to not reload the sqlalchemy modules, or if it has to, then get it to fully load sqlalchemy completely. I see that reload(mod) right in the stack trace there. On 6/26/14, 1:14 PM, Dustin Oprea wrote: I'm using SQLAlchemy with web.py, and have used it many times in the past. I'm working on a project using gevent/greenlets, and everything has been fine for a couple of months, until today. Suddenly, I'm getting these at a time when I'm not even receiving any requests. Does anyone have any advice or steps to troubleshoot? Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class
Re: [sqlalchemy] AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered
On Jun 26, 2014 1:34 PM, Mike Bayer mike...@zzzcomputing.com wrote: that looks pretty much like modules are being reloaded in process. Either get web.py to not reload the sqlalchemy modules, or if it has to, then get it to fully load sqlalchemy completely. I see that reload(mod) right in the stack trace there. Sure, but I have a dozen web.py projects, and gave been using it for a while, and haven't had this issue until yesterday. I thought that too, but doubt that's it. Dustin On 6/26/14, 1:14 PM, Dustin Oprea wrote: I'm using SQLAlchemy with web.py, and have used it many times in the past. I'm working on a project using gevent/greenlets, and everything has been fine for a couple of months, until today. Suddenly, I'm getting these at a time when I'm not even receiving any requests. Does anyone have any advice or steps to troubleshoot? Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File
Re: [sqlalchemy] Performance problem of simple orm usage
On 6/26/14, 12:24 PM, Mike Bayer wrote: I'm not sure what kind of application this is but I would not be mixing transactional control, that is the commit/rollback, inside of a business method that only seeks to create some new objects. There'd be a containing pattern within which store_enos_rec() is called these 20K times, then the commit() is called on the outside of that, or at least every 1000 records or so. Also, the create if not present pattern is often much better set up by preloading all the objects you'll deal with up front: existing_evs = dict(Session.query(EnosValues.enos_id, EnosValues)) if ev_id not in existing_evs: Session.add(EnosValues(...)) If you truly need to optimize a method whose job it is to start a new transaction, insert a single row if not present and then commit the transaction fully, I'd use a core insert() construct for that. Spinning up a whole unit of work in that context is overkill if performance is a concern. this is something that's pretty common and i may have the opportunity soon to come up with a helper for this case. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered
On 6/26/14, 1:49 PM, Dustin Oprea wrote: On Jun 26, 2014 1:34 PM, Mike Bayer mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote: that looks pretty much like modules are being reloaded in process.Either get web.py to not reload the sqlalchemy modules, or if it has to, then get it to fully load sqlalchemy completely. I see that reload(mod) right in the stack trace there. Sure, but I have a dozen web.py projects, and gave been using it for a while, and haven't had this issue until yesterday. I thought that too, but doubt that's it. I guarantee you that is it - maybe it was not being called on this particular module or combination of modules. module reloading is bad IMHO. Dustin On 6/26/14, 1:14 PM, Dustin Oprea wrote: I'm using SQLAlchemy with web.py, and have used it many times in the past. I'm working on a project using gevent/greenlets, and everything has been fine for a couple of months, until today. Suddenly, I'm getting these at a time when I'm not even receiving any requests. Does anyone have any advice or steps to troubleshoot? Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered % type_) AssertionError: Type class 'sqlalchemy.engine.interfaces.Connectable' is already registered Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/web/application.py, line 237, in process return p(lambda: process(processors)) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 565, in processor h() File /usr/local/lib/python2.7/dist-packages/web/application.py, line 661, in __call__ self.check(mod) File /usr/local/lib/python2.7/dist-packages/web/application.py, line 680, in check reload(mod) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 58, in module class Inspector(object): File /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/reflection.py, line 136, in Inspector @inspection._inspects(Connectable) File /usr/local/lib/python2.7/dist-packages/sqlalchemy/inspection.py, line 84, in decorate registered %
Re: [sqlalchemy] Performance problem of simple orm usage
On 6/26/14, 3:07 PM, Hans-Peter Jansen wrote:
Okay, attached is a profile dump with .commit() out of the way. Here's
the head of it: -- p = pstats.Stats('srelay.pstats') --
p.strip_dirs().sort_stats('cumulative').print_stats(100) Thu Jun 26
20:41:50 2014 srelay.pstats 55993702 function calls (54767154
primitive calls) in 533.479 seconds Ordered by: cumulative time List
reduced from 1277 to 100 due to restriction 100 ncalls tottime
percall cumtime percall filename:lineno(function) 1 0.000 0.000
533.523 533.523 cProfile.py:146(runcall) 1 0.174 0.174 533.523 533.523
srelay.py:23(run) 24558 1.060 0.000 458.633 0.019
srelay.py:74(store_enos_rec) 24586 0.111 0.000 434.802 0.018
query.py:2280(first) 24586 0.493 0.000 434.687 0.018
query.py:2143(__getitem__) 24634 0.114 0.000 431.682 0.018
query.py:2361(__iter__) 24683 1.330 0.000 391.984 0.016
session.py:1790(flush) 24634 0.052 0.000 391.950 0.016
session.py:1137(_autoflush) 21043 0.628 0.000 390.597 0.019
session.py:1841(_flush) 21043 0.396 0.000 385.326 0.018
unitofwork.py:350(execute) 21043 0.417 0.000 350.813 0.017
unitofwork.py:299(_generate_actions) 126514 0.814 0.000 348.487 0.003
unitofwork.py:411(execute) 189809 0.726 0.000 343.341 0.002
unitofwork.py:177(get_attribute_history) 21108 0.251 0.000 342.547
0.016 dependency.py:440(presort_saves) 42240 9.870 0.000 302.804 0.007
attributes.py:871(get_history) 42160 292.536 0.007 292.910 0.007
attributes.py:1321(from_collection) 45788 0.175 0.000 55.708 0.001
base.py:597(execute) 45735 0.410 0.000 53.341 0.001
base.py:727(_execute_clauseelement) 45788 0.511 0.000 41.410 0.001
base.py:812(_execute_context) 63209 39.185 0.001 39.294 0.001
attributes.py:1255(as_state) 2 0.001 0.000 39.039 19.519
process.py:83(__call__) 39 39.037 1.001 39.037 1.001 {time.sleep}
24634 0.157 0.000 36.100 0.001 query.py:2375(_execute_and_instances)
45788 0.098 0.000 34.287 0.001 default.py:323(do_execute) 45816 0.490
0.000 34.199 0.001 cursors.py:164(execute) 24562 0.513 0.000 32.848
0.001 fetch_srelay.py:57(fetch_srelay) 45816 0.083 0.000 32.323 0.001
cursors.py:353(_query) 45816 0.217 0.000 31.993 0.001
cursors.py:315(_do_query) 45816 30.949 0.001 30.949 0.001 {method
'query' of '_mysql.connection' objects} 198 0.001 0.000 30.394 0.154
fetch_srelay.py:44(recv) 396 0.003 0.000 30.393 0.077
fetch_srelay.py:33(recvall) 392 30.039 0.077 30.039 0.077 {method
'recv' of '_socket.socket' objects} 63126 0.203 0.000 26.584 0.000
unitofwork.py:522(execute) Unfortunately, it still suffers from about
the same runtime behavior.
Obviously, some operation triggers the flush method with about the
same consequences..
OK, turn off autoflush - either Session(autoflush=False), or within the
critical block, do with session.no_autoflush:
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
On Thu, Jun 26, 2014 at 5:57 AM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/25/14, 8:06 PM, Ken Lareau wrote:
On Wed, Jun 25, 2014 at 6:28 AM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/25/14, 2:26 AM, Ken Lareau wrote:
On Tue, Jun 24, 2014 at 3:35 PM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/23/14, 8:09 PM, Ken Lareau wrote:
if apptier:
subq = (
Session.query(
Package.pkg_name,
Package.version,
Package.revision,
AppDefinition.app_type,
AppDeployment.environment
).join(Deployment)
.join(AppDeployment)
.join(AppDefinition)
.filter(Package.pkg_name == package_name)
.filter(AppDeployment.environment == env)
.filter(AppDeployment.status != 'invalidated'))
[...]
# The actual column name must be used in the subquery
# usage below; DB itself should be corrected
versions = (Session.query(subq.c.appType,
subq.c.version,
subq.c.revision)
.group_by(subq.c.appType, subq.c.environment)
.all())
The parts in the subquery work fine, but the access of the 'column'
in the final query leads to this:
Traceback (most recent call last):
File stdin, line 1, in module
File /home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py,
line 234, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
This is not completely surprising, but I'm uncertain as the best way
to fix this... help? :)
nothing is obviously wrong, assuming you are calling subquery() on
subq at the end. It will be a select() construct which will have a
column called .environment on the .c. collection because
AppDeployment.environment is present. Can't say why this attribute is
not here without full details.
Oops there were some additional filters I left out, but this is added
before the final query:
subq = (subq.order_by(AppDeployment.realized.desc())
.subquery(name='t_ordered'))
Not sure why it isn't working, but today my coworker an I massively
rewrote one of the other methods to avoid this same issue, so maybe
we should try the same for this one... though it would be nice to know
what I goofed here, but not sure what additional information I can
give that would help offhand...
what are the names that are actually present on subq.c ? print
list(subq.c) should give an indication.
Adding that in to the method gave this:
deploy.find_deployed_version('tdstest', 'development', version='9',
apptier=True)
[Column('pkg_name', String(length=255), table=t_ordered,
nullable=False), Column('version', String(length=63), table=t_ordered,
nullable=False), Column('revision', String(length=63), table=t_ordered,
nullable=False), Column('appType', String(length=100), table=t_ordered,
nullable=False), sqlalchemy.sql.elements.ColumnClause at 0x1d20e10;
%(30521360 anon)s]
Traceback (most recent call last):
File stdin, line 1, in module
File /home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py, line
237, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
you need to put .label('environment') on that column before it finds its
way into subq. I dont have the mappings here to review.
for the next go around please just provide a one-file example of the
mapping plus the query, thanks.
Do you desire the example to be fully functional, or just have the relevant
mapping and query? We ran into the same issue with another query and
ended up completely reworking the code to avoid the subquery, but it in-
volves quite a bit of various mappings and I'm uncertain if it's the proper
way to move forward. I could try to post the before/after here, but if it
needs to be able to to run standalone it may take me a bit of time to get
it into a fully workable state (given how extensive it is).
--
- Ken Lareau
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
minimal mapping + the query against that mapping.
On 6/26/14, 4:11 PM, Ken Lareau wrote:
On Thu, Jun 26, 2014 at 5:57 AM, Mike Bayer mike...@zzzcomputing.com
mailto:mike...@zzzcomputing.com wrote:
On 6/25/14, 8:06 PM, Ken Lareau wrote:
On Wed, Jun 25, 2014 at 6:28 AM, Mike Bayer
mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote:
On 6/25/14, 2:26 AM, Ken Lareau wrote:
On Tue, Jun 24, 2014 at 3:35 PM, Mike Bayer
mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com
wrote:
On 6/23/14, 8:09 PM, Ken Lareau wrote:
if apptier:
subq = (
Session.query(
Package.pkg_name,
Package.version,
Package.revision,
AppDefinition.app_type,
AppDeployment.environment
).join(Deployment)
.join(AppDeployment)
.join(AppDefinition)
.filter(Package.pkg_name == package_name)
.filter(AppDeployment.environment == env)
.filter(AppDeployment.status !=
'invalidated'))
[...]
# The actual column name must be used in the
subquery
# usage below; DB itself should be corrected
versions = (Session.query(subq.c.appType,
subq.c.version,
subq.c.revision)
.group_by(subq.c.appType,
subq.c.environment)
.all())
The parts in the subquery work fine, but the access of
the 'column'
in the final query leads to this:
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py,
line 234, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
This is not completely surprising, but I'm uncertain
as the best way
to fix this... help? :)
nothing is obviously wrong, assuming you are calling
subquery() on
subq at the end. It will be a select() construct which
will have a
column called .environment on the .c. collection because
AppDeployment.environment is present. Can't say why
this attribute is
not here without full details.
Oops there were some additional filters I left out, but this
is added
before the final query:
subq = (subq.order_by(AppDeployment.realized.desc())
.subquery(name='t_ordered'))
Not sure why it isn't working, but today my coworker an I
massively
rewrote one of the other methods to avoid this same issue,
so maybe
we should try the same for this one... though it would be
nice to know
what I goofed here, but not sure what additional information
I can
give that would help offhand...
what are the names that are actually present on subq.c ?
print list(subq.c) should give an indication.
Adding that in to the method gave this:
deploy.find_deployed_version('tdstest', 'development',
version='9', apptier=True)
[Column('pkg_name', String(length=255), table=t_ordered,
nullable=False), Column('version', String(length=63),
table=t_ordered, nullable=False), Column('revision',
String(length=63), table=t_ordered, nullable=False),
Column('appType', String(length=100), table=t_ordered,
nullable=False), sqlalchemy.sql.elements.ColumnClause at
0x1d20e10; %(30521360 anon)s]
Traceback (most recent call last):
File stdin, line 1, in module
File
/home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py,
line 237, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
you need to put .label('environment') on that column before it
finds its way into subq. I dont have
Re: [sqlalchemy] Performance problem of simple orm usage
Dear Mike,
sorry for not coping with preferred reply behavior..
On Donnerstag, 26. Juni 2014 15:26:02 Mike Bayer wrote:
On 6/26/14, 3:07 PM, Hans-Peter Jansen wrote:
Obviously, some operation triggers the flush method with about the
same consequences..
OK, turn off autoflush - either Session(autoflush=False), or within the
critical block, do with session.no_autoflush:
Okay, that makes a difference, shaving of about 40% of total runtime,
after dealing with identical records in one session..
Still, attributes is highest in profiles. I guess, this is, what you
called overkill in your first response. A helper for dealing with this
situation sounds interesting, though.
Thu Jun 26 20:41:50 2014srelay.pstats
55993702 function calls (54767154 primitive calls) in 533.479 seconds
Ordered by: internal time
List reduced from 1277 to 30 due to restriction 30
ncalls tottime percall cumtime percall filename:lineno(function)
42160 292.5360.007 292.9100.007
attributes.py:1321(from_collection)
63209 39.1850.001 39.2940.001 attributes.py:1255(as_state)
39 39.0371.001 39.0371.001 {time.sleep}
45816 30.9490.001 30.9490.001 {method 'query' of
'_mysql.connection' objects}
392 30.0390.077 30.0390.077 {method 'recv' of
'_socket.socket' objects}
422409.8700.000 302.8040.007 attributes.py:871(get_history)
210834.2840.0006.5000.000 attributes.py:868(__copy)
421582.2300.0002.2520.000 collections.py:711(__len__)
210431.4660.0001.8510.000 topological.py:51(find_cycles)
41594301.3690.0001.3690.000 {isinstance}
246831.3300.000 391.9840.016 session.py:1790(flush)
642077/247121.2920.000 12.8400.001
visitors.py:74(_compiler_dispatch)
1975701.2850.0004.2150.000 compiler.py:389(visit_label)
457351.2350.0003.6150.000 default.py:391(_init_compiled)
2468471.1450.0001.6370.000 compiler.py:421(visit_column)
245581.0600.000 458.6330.019 srelay.py:74(store_enos_rec)
246361.0480.000 12.7090.001 compiler.py:1136(visit_select)
1975521.0300.0006.0000.000
compiler.py:1019(_label_select_column)
246361.0150.0001.7930.000 result.py:171(__init__)
631260.9390.0003.2020.000
persistence.py:275(_collect_update_commands)
49200/491720.9370.0003.5390.000 db_scheme.py:91(__repr__)
1472400.9040.0001.2460.000 topological.py:15(sort_as_subsets)
1265140.8140.000 348.4870.003 unitofwork.py:411(execute)
631910.7710.0001.9840.000
dependency.py:67(per_property_flush_actions)
6229250.7510.0000.9730.000 attributes.py:308(__get__)
6996820.7450.0000.7450.000 state.py:185(dict)
1898090.7260.000 343.3410.002
unitofwork.py:177(get_attribute_history)
739280.6530.0000.7810.000 expression.py:3538(__init__)
421700.6480.0000.6480.000 {method 'clear' of 'dict' objects}
1262520.6390.0001.7950.000 persistence.py:802(_sort_states)
Cheers,
Pete
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Performance problem of simple orm usage
On 6/26/14, 5:18 PM, Hans-Peter Jansen wrote:
Dear Mike,
sorry for not coping with preferred reply behavior..
On Donnerstag, 26. Juni 2014 15:26:02 Mike Bayer wrote:
On 6/26/14, 3:07 PM, Hans-Peter Jansen wrote:
Obviously, some operation triggers the flush method with about the
same consequences..
OK, turn off autoflush - either Session(autoflush=False), or within the
critical block, do with session.no_autoflush:
Okay, that makes a difference, shaving of about 40% of total runtime,
after dealing with identical records in one session..
Still, attributes is highest in profiles. I guess, this is, what you
called overkill in your first response. A helper for dealing with this
situation sounds interesting, though.
Thu Jun 26 20:41:50 2014srelay.pstats
55993702 function calls (54767154 primitive calls) in 533.479 seconds
Ordered by: internal time
List reduced from 1277 to 30 due to restriction 30
ncalls tottime percall cumtime percall filename:lineno(function)
42160 292.5360.007 292.9100.007
attributes.py:1321(from_collection)
63209 39.1850.001 39.2940.001 attributes.py:1255(as_state)
from_collection is essentially returning a structure that describes
which members of a collection were removed or added. if the collection
is totally empty or is unloaded, this method should be much faster. try
not do to this:
enos.values.append(ev)
that's what's causing that latency to occur. The one-to-many of
enos-ev is mirrored by a many to one from ev-enos - set enos as a
parent of ev instead, or even faster, just set the foreign key column on
ev to that of the primary key of enos.
39 39.0371.001 39.0371.001 {time.sleep}
45816 30.9490.001 30.9490.001 {method 'query' of
'_mysql.connection' objects}
392 30.0390.077 30.0390.077 {method 'recv' of
'_socket.socket' objects}
422409.8700.000 302.8040.007 attributes.py:871(get_history)
210834.2840.0006.5000.000 attributes.py:868(__copy)
421582.2300.0002.2520.000 collections.py:711(__len__)
210431.4660.0001.8510.000 topological.py:51(find_cycles)
41594301.3690.0001.3690.000 {isinstance}
246831.3300.000 391.9840.016 session.py:1790(flush)
642077/247121.2920.000 12.8400.001
visitors.py:74(_compiler_dispatch)
1975701.2850.0004.2150.000 compiler.py:389(visit_label)
457351.2350.0003.6150.000 default.py:391(_init_compiled)
2468471.1450.0001.6370.000 compiler.py:421(visit_column)
245581.0600.000 458.6330.019 srelay.py:74(store_enos_rec)
246361.0480.000 12.7090.001 compiler.py:1136(visit_select)
1975521.0300.0006.0000.000
compiler.py:1019(_label_select_column)
246361.0150.0001.7930.000 result.py:171(__init__)
631260.9390.0003.2020.000
persistence.py:275(_collect_update_commands)
49200/491720.9370.0003.5390.000 db_scheme.py:91(__repr__)
1472400.9040.0001.2460.000
topological.py:15(sort_as_subsets)
1265140.8140.000 348.4870.003 unitofwork.py:411(execute)
631910.7710.0001.9840.000
dependency.py:67(per_property_flush_actions)
6229250.7510.0000.9730.000 attributes.py:308(__get__)
6996820.7450.0000.7450.000 state.py:185(dict)
1898090.7260.000 343.3410.002
unitofwork.py:177(get_attribute_history)
739280.6530.0000.7810.000 expression.py:3538(__init__)
421700.6480.0000.6480.000 {method 'clear' of 'dict'
objects}
1262520.6390.0001.7950.000 persistence.py:802(_sort_states)
Cheers,
Pete
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Performance problem of simple orm usage
In case this helps... This reminds me slightly of some RFID work I did years ago. We had a lot of reads coming in from different units, several reads per unit per second. I found the best way to handle writing was to just access the db directly, but kept the ORM on the read side. I recall a few things from that experience. some of this might be applicable to you: • doing a Select else Insert pattern was faster than Insert All and just letting the db constraint catch errors. there were still a lot of exceptions from different transactions. this is probably dependent on the database/version/size/etc. but it's worth profiling. • we had a low number of devices, so just cached their data into the app's memory. losing the extra select helped a bit. * our transaction got pared down to looking like this: begin; select record ; insert record; commit; * i think someone eventually migrated the db interaction to use a stored procedure. * we kept an in-memory array of recently seen transactions. it was just a simple list that would have elements inserted at 0 and then truncated. we checked that list first, then hit the database if it wasn't seen . our performance greatly increased, and frustrations decreased, as we moved more logic out of the database and into python. on another RFID project, the lead engineer i worked with had a similar approach... however to keep performance better, he used an in-memory database to record data and then did periodic flushes to the real database after the data was smoothed out ( to compensate for known hardware issues ). -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
On Thu, Jun 26, 2014 at 1:45 PM, Mike Bayer mike...@zzzcomputing.com
wrote:
minimal mapping + the query against that mapping.
Okay, for the other method I needed to change, I think I've put together
complete enough examples and have attached the files. The 'before'
file is what we have currently (and doesn't quite do the right thing as
it's only using the environment_id), and the 'after' file is what seemingly
worked after my coworker and I tried a different route to avoid having
to use a subquery. One thing to note is there's a reciprocating rela-
tionship added on the Deployment class to the Package class which
initially caused a lovely stack overflow in Python until the joinedload
options were added; this may be a bad idea, but it was the only way
we knew how to get things working the way we needed. :)
I tried to add all the imports as well in case there were any questions
that might come from the code itself; hopefully I didn't miss any, but
if there's still confusion, let me know.
- Ken
On 6/26/14, 4:11 PM, Ken Lareau wrote:
On Thu, Jun 26, 2014 at 5:57 AM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/25/14, 8:06 PM, Ken Lareau wrote:
On Wed, Jun 25, 2014 at 6:28 AM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/25/14, 2:26 AM, Ken Lareau wrote:
On Tue, Jun 24, 2014 at 3:35 PM, Mike Bayer mike...@zzzcomputing.com
wrote:
On 6/23/14, 8:09 PM, Ken Lareau wrote:
if apptier:
subq = (
Session.query(
Package.pkg_name,
Package.version,
Package.revision,
AppDefinition.app_type,
AppDeployment.environment
).join(Deployment)
.join(AppDeployment)
.join(AppDefinition)
.filter(Package.pkg_name == package_name)
.filter(AppDeployment.environment == env)
.filter(AppDeployment.status != 'invalidated'))
[...]
# The actual column name must be used in the subquery
# usage below; DB itself should be corrected
versions = (Session.query(subq.c.appType,
subq.c.version,
subq.c.revision)
.group_by(subq.c.appType, subq.c.environment)
.all())
The parts in the subquery work fine, but the access of the 'column'
in the final query leads to this:
Traceback (most recent call last):
File stdin, line 1, in module
File /home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py,
line 234, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
This is not completely surprising, but I'm uncertain as the best way
to fix this... help? :)
nothing is obviously wrong, assuming you are calling subquery() on
subq at the end. It will be a select() construct which will have a
column called .environment on the .c. collection because
AppDeployment.environment is present. Can't say why this attribute is
not here without full details.
Oops there were some additional filters I left out, but this is added
before the final query:
subq = (subq.order_by(AppDeployment.realized.desc())
.subquery(name='t_ordered'))
Not sure why it isn't working, but today my coworker an I massively
rewrote one of the other methods to avoid this same issue, so maybe
we should try the same for this one... though it would be nice to know
what I goofed here, but not sure what additional information I can
give that would help offhand...
what are the names that are actually present on subq.c ? print
list(subq.c) should give an indication.
Adding that in to the method gave this:
deploy.find_deployed_version('tdstest', 'development', version='9',
apptier=True)
[Column('pkg_name', String(length=255), table=t_ordered,
nullable=False), Column('version', String(length=63), table=t_ordered,
nullable=False), Column('revision', String(length=63), table=t_ordered,
nullable=False), Column('appType', String(length=100), table=t_ordered,
nullable=False), sqlalchemy.sql.elements.ColumnClause at 0x1d20e10;
%(30521360 anon)s]
Traceback (most recent call last):
File stdin, line 1, in module
File /home/klareau/repos/git/tagopsdb/tagopsdb/deploy/deploy.py, line
237, in find_deployed_version
.group_by(subq.c.appType, subq.c.environment)
File
/home/klareau/.virtualenvs/tds/lib/python2.6/site-packages/sqlalchemy/util/_collections.py,
line 174, in __getattr__
raise AttributeError(key)
AttributeError: environment
you need to put .label('environment') on that column before it finds
its way into subq. I dont have the mappings here to review.
for the next go around please just provide a one-file example of
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
a self contained version of before is attached, seems to work as is (works in 0.8 too). Modify it to show me it not working. output: SELECT t_ordered.pkg_name AS t_ordered_pkg_name, t_ordered.version AS t_ordered_version, t_ordered.revision AS t_ordered_revision, t_ordered.appType AS t_ordered_appType, t_ordered.environment_id AS t_ordered_environment_id, t_ordered.realized AS t_ordered_realized, t_ordered.user AS t_ordered_user, t_ordered.status AS t_ordered_status FROM (SELECT packages.pkg_name AS pkg_name, packages.version AS version, packages.revision AS revision, app_definitions.appType AS appType, app_deployments.environment_id AS environment_id, (SELECT environments.environment FROM environments WHERE environments.environmentID = app_deployments.environment_id) AS anon_1, app_deployments.realized AS realized, app_deployments.user AS user, app_deployments.status AS status FROM packages JOIN deployments ON packages.package_id = deployments.package_id JOIN app_deployments ON deployments.DeploymentID = app_deployments.DeploymentID JOIN app_definitions ON app_definitions.AppID = app_deployments.AppID WHERE app_deployments.status != :status_1 AND (SELECT environments.environment FROM environments WHERE environments.environmentID = app_deployments.environment_id) = :param_1 ORDER BY app_deployments.realized DESC) AS t_ordered GROUP BY t_ordered.appType, t_ordered.environment_id, t_ordered.pkg_name HAVING t_ordered.status LIKE :status_2 AND unix_timestamp(t_ordered.realized) unix_timestamp(now()) - :unix_timestamp_1 On 6/26/14, 7:41 PM, Ken Lareau wrote: On Thu, Jun 26, 2014 at 1:45 PM, Mike Bayer mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote: minimal mapping + the query against that mapping. Okay, for the other method I needed to change, I think I've put together complete enough examples and have attached the files. The 'before' file is what we have currently (and doesn't quite do the right thing as it's only using the environment_id), and the 'after' file is what seemingly worked after my coworker and I tried a different route to avoid having to use a subquery. One thing to note is there's a reciprocating rela- tionship added on the Deployment class to the Package class which initially caused a lovely stack overflow in Python until the joinedload options were added; this may be a bad idea, but it was the only way we knew how to get things working the way we needed. :) I tried to add all the imports as well in case there were any questions that might come from the code itself; hopefully I didn't miss any, but if there's still confusion, let me know. - Ken On 6/26/14, 4:11 PM, Ken Lareau wrote: On Thu, Jun 26, 2014 at 5:57 AM, Mike Bayer mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote: On 6/25/14, 8:06 PM, Ken Lareau wrote: On Wed, Jun 25, 2014 at 6:28 AM, Mike Bayer mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote: On 6/25/14, 2:26 AM, Ken Lareau wrote: On Tue, Jun 24, 2014 at 3:35 PM, Mike Bayer mike...@zzzcomputing.com mailto:mike...@zzzcomputing.com wrote: On 6/23/14, 8:09 PM, Ken Lareau wrote: if apptier: subq = ( Session.query( Package.pkg_name, Package.version, Package.revision, AppDefinition.app_type, AppDeployment.environment ).join(Deployment) .join(AppDeployment) .join(AppDefinition) .filter(Package.pkg_name == package_name) .filter(AppDeployment.environment == env) .filter(AppDeployment.status != 'invalidated')) [...] # The actual column name must be used in the subquery # usage below; DB itself should be corrected versions = (Session.query(subq.c.appType, subq.c.version, subq.c.revision) .group_by(subq.c.appType, subq.c.environment) .all()) The parts in the subquery work fine, but the access of the 'column' in the final query leads to this: Traceback (most recent call last): File stdin, line 1, in module File
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
Done, new file attached (this gives the same error message as the one I
showed initially, at least on my system).
- Ken
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
from sqlalchemy import *
from sqlalchemy.dialects.mysql import INTEGER, SMALLINT, TIMESTAMP
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
import datetime
Base = declarative_base()
class AppDefinition(Base):
__tablename__ = 'app_definitions'
id = Column(u'AppID', SMALLINT(display_width=2), primary_key=True)
app_type = Column(u'appType', String(length=100), nullable=False)
status = Column(
Enum('active', 'inactive'),
nullable=False,
server_default='active'
)
app_deployments = relationship('AppDeployment')
class AppDeployment(Base):
__tablename__ = 'app_deployments'
id = Column(u'AppDeploymentID', INTEGER(), primary_key=True)
deployment_id = Column(
u'DeploymentID',
INTEGER(),
ForeignKey('deployments.DeploymentID', ondelete='cascade'),
nullable=False
)
app_id = Column(
u'AppID',
SMALLINT(display_width=6),
ForeignKey('app_definitions.AppID', ondelete='cascade'),
nullable=False
)
user = Column(String(length=32), nullable=False)
status = Column(
Enum(
'complete',
'incomplete',
'inprogress',
'invalidated',
'validated',
),
nullable=False
)
environment_id = Column(
u'environment_id',
INTEGER(),
ForeignKey('environments.environmentID', ondelete='cascade'),
nullable=False
)
realized = Column(
TIMESTAMP(),
nullable=False,
server_default=func.current_timestamp()
)
application = relationship('AppDefinition')
deployment = relationship('Deployment')
environment_obj = relationship('Environment')
@hybrid_property
def environment(self):
return self.environment_obj.environment
@environment.expression
def environment(cls):
return select(
[Environment.environment]
).where(
Environment.id == cls.environment_id
).correlate(cls).as_scalar()
class Deployment(Base):
__tablename__ = 'deployments'
id = Column(u'DeploymentID', INTEGER(), primary_key=True)
package_id = Column(
INTEGER(),
ForeignKey('packages.package_id', ondelete='cascade'),
nullable=False
)
app_deployments = relationship('AppDeployment')
class Environment(Base):
__tablename__ = 'environments'
id = Column(u'environmentID', INTEGER(), primary_key=True)
environment = Column(String(length=15), nullable=False, unique=True)
class Package(Base):
__tablename__ = 'packages'
id = Column(u'package_id', INTEGER(), primary_key=True)
pkg_name = Column(String(length=255), nullable=False)
version = Column(String(length=63), nullable=False)
revision = Column(String(length=63), nullable=False)
deployments = relationship('Deployment')
Session = scoped_session(sessionmaker())
subq = (Session.query(Package.pkg_name, Package.version,
Package.revision, AppDefinition.app_type,
AppDeployment.environment,
AppDeployment.realized, AppDeployment.user,
AppDeployment.status)
.join(Deployment)
.join(AppDeployment)
.join(AppDefinition)
.filter(AppDeployment.status != 'invalidated')
.filter(AppDeployment.environment == some environment)
.order_by(AppDeployment.realized.desc())
.subquery(name='t_ordered'))
q = (Session.query(subq.c.pkg_name, subq.c.version, subq.c.revision,
subq.c.appType, subq.c.environment,
subq.c.realized, subq.c.user, subq.c.status)
.group_by(subq.c.appType, subq.c.environment_id,
subq.c.pkg_name)
.having(and_(subq.c.status.like('%complete'),
func.unix_timestamp(subq.c.realized)
func.unix_timestamp(func.now()) -
datetime.timedelta(seconds=60)))
)
print q
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
right, so a few emails ago I said:
you need to put .label('environment') on that column before it finds
its way into subq. I dont have the mappings here to review.
here's that:
@environment.expression
def environment(cls):
return select(
[Environment.environment]
).where(
Environment.id == cls.environment_id
).correlate(cls).label('environment')
On 6/26/14, 9:50 PM, Ken Lareau wrote:
Done, new file attached (this gives the same error message as the one
I showed initially, at least on my system).
- Ken
--
You received this message because you are subscribed to the Google
Groups sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
mailto:sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com
mailto:sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] A question regarding hybrid properties, relationships and schema changes
On Jun 26, 2014 7:40 PM, Mike Bayer mike...@zzzcomputing.com wrote:
right, so a few emails ago I said:
you need to put .label('environment') on that column before it finds
its way into subq. I dont have the mappings here to review.
here's that:
@environment.expression
def environment(cls):
return select(
[Environment.environment]
).where(
Environment.id == cls.environment_id
).correlate(cls).label('environment')
Aha... thanks. Now I'm afraid to ask if you looked at the after file,
and if so, considered it insane. :)
- Ken
On 6/26/14, 9:50 PM, Ken Lareau wrote:
Done, new file attached (this gives the same error message as the one I
showed initially, at least on my system).
- Ken
--
You received this message because you are subscribed to the Google
Groups sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an
email to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
