Re: [sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
From my understanding, this should be safe with Twisted: - SqlAlchemy engine operations, where you don't use the ORM and encapsulate every ansync chunk in a transaction, and within a thread - SqlAlchemy orm operations, where every async chunk is encapsulated in it's own session, and within a thread. I'm not sure how familiar you are with twisted, so i'm going to go real basic here... In the way you'd likely use it, Twisted works on the concept of async 'deferreds'. You sort of play Choose your own adventure and add in Callbacks or Errbacks to your miscellaneous functions. Processing a request wouldn't be: def process_request(): hit_api blocked process_results it would be more like (and I'm butchering this syntax): def process_request(): d = deferred( get_api ).addCallback(got_api) d.deferToThread() def get_api() block def got_api(): process data You'd also probably use a DeferredList, which lets you queue up 2dozen items in a batch at once. it's rad. Anyways, you wouldn't want to use a SqlAlchemy session that persists throughout the callback chain. Because everything is so anync , you're virtually guaranteed that the Session will not match the datastore. SqlAlchemy also has a blocking interface to the datastore, so IIRC, you not have issues with all the sessions, but you end up blocking the main twisted thread. so the popular workaround(s) are to do both: 1- defer to thread (which most people would do anyways) 2- open/close multiple sessions (which are cheap). basically , instead of approaching a task in twisted as a single action of multiple components, think of it like a website , with each action as a new page request. your web session is the same, but your SqlAlchemy Session changes on each request. Anyways, I would probably look more at Celery for what you're looking to accomplish. Celery is super simple to learn and maintain; On a learning-curve scale of 1-10, I'd probably put celery at 1, gevent at 5, and twisted at 10. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
On Mon, Oct 7, 2013 at 12:01 PM, Jonathan Vanasco jonat...@findmeon.com wrote: so the popular workaround(s) are to do both: 1- defer to thread (which most people would do anyways) 2- open/close multiple sessions (which are cheap). basically , instead of approaching a task in twisted as a single action of multiple components, think of it like a website , with each action as a new page request. your web session is the same, but your SqlAlchemy Session changes on each request. There's a 3. 3- open a single request-scope session, and carry it through your async tasks. You'll have to be super-careful not to use any thread-local state, always use the request-scope session, but that's relatively easy with most frameworks. Probably the easiest way would be to instantiate an explicit session with a session_maker, and then carry it implicitly in your callbacks as a closure. This is even easier on web frameworks, that support this kind of stuff built-in since it's so very common. But, adapting, got_api: def process_request(): d = deferToThread( get_api, session_maker() ).addCallback(got_api) def get_api(session) # use session block def got_api(): process data At first sight, this doesn't look all that better, but when you start chaining deferreds, it will be. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
On Monday, October 7, 2013 8:27:34 PM UTC-4, Klauss wrote: 3- open a single request-scope session, and carry it through your async tasks. I left that out, because it brings up the problem where your long-running session blocks the database (via transactions) or if you're in some 'autocommit' mode, you end up with a session very much out of sync. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
Thanks guys. Jonathan, that is, I think, what will be going on. So, I'm not quite clear from your comments, using SQLAlchemy with Twisted is practical now or not? Is it better to do it with Twisted than with Gevent? If you have the time to explain further about the persisting the session comment that would be great. At the moment I'm just trying to figure out what I want to figure out. ;-) thanks Iain On Thu, Oct 3, 2013 at 5:05 PM, Jonathan Vanasco jonat...@findmeon.comwrote: Also, a decent strategy to use is this: a) Your client does a lot of business logic in Pyramid. When you need to bill you hit an internal API or create a celery task b) You render a waiting page, and have ajax, or refresh, check to see if the internal API , or celery, is done processing c) when the internal api is done processing, you show the thank you screen. getting my SqlAlchemy model to work on Celery was a bit of a nightmare. i can try and dig up exactly what I did. IIRC, the problem was the Celery didn't seem to have an 'on initialization' routine, so I had to do some janky shit call my sqlalchemy initialization stuff. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
[sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
So the No SqlAlchemy with Twisted approach is dated. I've been on the twisted list for a while, and recently asked it -- as i have a bit of stuff for the same project running in twisted as-well-as pyramid and-also celery. The current consensus is that it's a general no-go/bad-idea if you're needing to persist the session; but you're going to run into that problem with any technology Check out the responses in this thread: July - http://twistedmatrix.com/pipermail/twisted-python/2013-July/027203.html August http://twistedmatrix.com/pipermail/twisted-python/2013-August/027333.html I resurrected it in August, and thats where some people started chiming in with ways SqlAlchemy can play nice. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
[sqlalchemy] Re: using SQLA in distributed, concurrent, or asynchronous contexts?
Also, a decent strategy to use is this: a) Your client does a lot of business logic in Pyramid. When you need to bill you hit an internal API or create a celery task b) You render a waiting page, and have ajax, or refresh, check to see if the internal API , or celery, is done processing c) when the internal api is done processing, you show the thank you screen. getting my SqlAlchemy model to work on Celery was a bit of a nightmare. i can try and dig up exactly what I did. IIRC, the problem was the Celery didn't seem to have an 'on initialization' routine, so I had to do some janky shit call my sqlalchemy initialization stuff. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
