[sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?
Hi folks, I just got a new position and am hoping I can advocate for using SQLAlchemy there. I will be doing some backend web service stuff that interacts with a db, but likely in some form of distributed architecture. It's all going to be on AWS. I believe there will be some service implementation that either needs to be highly performant or highly concurrent or some combination of both. IE there may be lots of clients hitting the service and the service has to interact with third party payment services which might be slow, and we don't want the server choking just waiting on the third party responses. Up till now I haven't had to do stuff in this domain. Can anyone tell me what I might want to look into as far as ways of handling concurrency or non-blocking services that will play fair with SQLAlchemy? I read that trying to get SA working on Twisted is not a good plan. Would love any pointers on what to read up on, use, etc. Right now all my experience is on Pylons/Pyramid and has not had to use any fancy messaging, deferred processing, or concurreny handling so I think I have a lot reading ahead of me. Thanks! Iain -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?
well there's always gevent. It might even be possible to get twisted and gevent to work together. So far I've managed to stick with multiprocessing for any kind of parallelism and it's done fine, but I'm not scaling up to hundreds or thousands of simultaneous things going on. On Oct 3, 2013, at 2:23 PM, Iain Duncan iainduncanli...@gmail.com wrote: Hi folks, I just got a new position and am hoping I can advocate for using SQLAlchemy there. I will be doing some backend web service stuff that interacts with a db, but likely in some form of distributed architecture. It's all going to be on AWS. I believe there will be some service implementation that either needs to be highly performant or highly concurrent or some combination of both. IE there may be lots of clients hitting the service and the service has to interact with third party payment services which might be slow, and we don't want the server choking just waiting on the third party responses. Up till now I haven't had to do stuff in this domain. Can anyone tell me what I might want to look into as far as ways of handling concurrency or non-blocking services that will play fair with SQLAlchemy? I read that trying to get SA working on Twisted is not a good plan. Would love any pointers on what to read up on, use, etc. Right now all my experience is on Pylons/Pyramid and has not had to use any fancy messaging, deferred processing, or concurreny handling so I think I have a lot reading ahead of me. Thanks! Iain -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [sqlalchemy] using SQLA in distributed, concurrent, or asynchronous contexts?
On Thu, Oct 3, 2013 at 6:02 PM, Michael Bayer mike...@zzzcomputing.com wrote: well there's always gevent. It might even be possible to get twisted and gevent to work together. So far I've managed to stick with multiprocessing for any kind of parallelism and it's done fine, but I'm not scaling up to hundreds or thousands of simultaneous things going on. On Oct 3, 2013, at 2:23 PM, Iain Duncan iainduncanli...@gmail.com wrote: Hi folks, I just got a new position and am hoping I can advocate for using SQLAlchemy there. I will be doing some backend web service stuff that interacts with a db, but likely in some form of distributed architecture. It's all going to be on AWS. I believe there will be some service implementation that either needs to be highly performant or highly concurrent or some combination of both. IE there may be lots of clients hitting the service and the service has to interact with third party payment services which might be slow, and we don't want the server choking just waiting on the third party responses. Up till now I haven't had to do stuff in this domain. Can anyone tell me what I might want to look into as far as ways of handling concurrency or non-blocking services that will play fair with SQLAlchemy? I read that trying to get SA working on Twisted is not a good plan. It depends a lot on the workloads. I've tried gevent with pure SQLA apps, and it works quite dandy. However, as soon as you start adding the stuff all applications are made of (libraries), you risk breakage more and more. I've found out the hard way. As in segfaults. I think most relatively high-level pure-python libs should do fine. Mixing multiprocessing with gevent, doesn't do fine. Trow in other reactors (zmq on threads in my case), and things also break pretty badly. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/groups/opt_out.