Re: [sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?

2013-10-03 Thread Michael Bayer

well there's always gevent.  It might even be possible to get twisted and 
gevent to work together.

So far I've managed to stick with multiprocessing for any kind of parallelism 
and it's done fine, but I'm not scaling up to hundreds or thousands of 
simultaneous things going on.



On Oct 3, 2013, at 2:23 PM, Iain Duncan iainduncanli...@gmail.com wrote:

 Hi folks, I just got a new position and am hoping I can advocate for using 
 SQLAlchemy there. I will be doing some backend web service stuff that 
 interacts with a db, but likely in some form of distributed architecture. 
 It's all going to be on AWS. I believe there will be some service 
 implementation that either needs to be highly performant or highly concurrent 
 or some combination of both. IE there may be lots of clients hitting the 
 service and the service has to interact with third party payment services 
 which might be slow, and we don't want the server choking just waiting on the 
 third party responses.
 
 Up till now I haven't had to do stuff in this domain. Can anyone tell me what 
 I might want to look into as far as ways of handling concurrency or 
 non-blocking services that will play fair with SQLAlchemy? I read that trying 
 to get SA working on Twisted is not a good plan.
 
 Would love any pointers on what to read up on, use, etc. Right now all my 
 experience is on Pylons/Pyramid and has not had to use any fancy messaging, 
 deferred processing, or concurreny handling so I think I have a lot reading 
 ahead of me.
 
 Thanks!
 Iain
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 sqlalchemy group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to sqlalchemy+unsubscr...@googlegroups.com.
 To post to this group, send email to sqlalchemy@googlegroups.com.
 Visit this group at http://groups.google.com/group/sqlalchemy.
 For more options, visit https://groups.google.com/groups/opt_out.



signature.asc
Description: Message signed with OpenPGP using GPGMail


Re: [sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?

2013-10-03 Thread Claudio Freire
On Thu, Oct 3, 2013 at 6:02 PM, Michael Bayer mike...@zzzcomputing.com wrote:

 well there's always gevent.  It might even be possible to get twisted and
 gevent to work together.

 So far I've managed to stick with multiprocessing for any kind of
 parallelism and it's done fine, but I'm not scaling up to hundreds or
 thousands of simultaneous things going on.



 On Oct 3, 2013, at 2:23 PM, Iain Duncan iainduncanli...@gmail.com wrote:

 Hi folks, I just got a new position and am hoping I can advocate for using
 SQLAlchemy there. I will be doing some backend web service stuff that
 interacts with a db, but likely in some form of distributed architecture.
 It's all going to be on AWS. I believe there will be some service
 implementation that either needs to be highly performant or highly
 concurrent or some combination of both. IE there may be lots of clients
 hitting the service and the service has to interact with third party payment
 services which might be slow, and we don't want the server choking just
 waiting on the third party responses.

 Up till now I haven't had to do stuff in this domain. Can anyone tell me
 what I might want to look into as far as ways of handling concurrency or
 non-blocking services that will play fair with SQLAlchemy? I read that
 trying to get SA working on Twisted is not a good plan.


It depends a lot on the workloads.

I've tried gevent with pure SQLA apps, and it works quite dandy.

However, as soon as you start adding the stuff all applications are
made of (libraries), you risk breakage more and more. I've found out
the hard way. As in segfaults.

I think most relatively high-level pure-python libs should do fine.
Mixing multiprocessing with gevent, doesn't do fine. Trow in other
reactors (zmq on threads in my case), and things also break pretty
badly.

-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.