On Mon, Feb 7, 2011 at 11:08 AM, Jed Smith <[email protected]> wrote: > On Mon, Feb 7, 2011 at 1:03 PM, Tomaz Muraus <[email protected]> wrote: >> but Twited has enough of syntactic sugar which makes it a lot less painful > > I still consider asynchronous a bad idea, but I consider Twisted even worse. > > I will strongly -1 any Twisted integration.
Lets slow down, I think we are getting caught up by other things. Right now this just feels like witch burning of Twisted Python (and don't get me wrong, I have plenty of issues with Twisted Python) I think we could implement asynchronous support in libcloud in a manner that was non-disruptive and without breaking the existing public API. I think if we look hard at our existing compute drivers, we see several common patterns, which in some places we have well abstracted, in others not as much. The pattern I see in list_node (and I think most other operations): 1) Preparing the request of information. (Driver.list_node) 2) Making the request (Connection.request) 3) Parsing the results of that request. (Response.parse_body, Driver.list_node The Rackspace driver (speaking of it because its complicated, not because of my (new) employer) is slightly harder because of how it gets its authentication token. But the core pattern still holds in my eyes. Thinking abstractly, where async lives in this place, is purely in abstraction of how step 2, Making the request, is done. If a driver was changed to just get a callback with the results, it could work in both an async and non-async version with absolutely no differences inside the actual drivers. I definitely agree that that this kind of abstraction is far more important for for storage, where common operations are to upload 10gb files -- or even consider the new S3 API that lets you upload multiple chunks in parallel -- doing that with a blocking API just doesn't make sense. I see a push to break up steps 1, 2, and 3 from each other also makes it much easier to write test cases, because you can more directly test the parsing of the result in step 3, and because you can more easily test if the request is constructed correctly in step 1. I'm not saying there is one solution to this, but lets just keep the mind open before throwing around -1s before any technical implementation beyond a very raw proposal has been done -- none of even know how it would work right now. My own personal vision is that an optional dependency on Twisted Python is a good thing, it does expose you to more environments. I agreee adding a hard dependency on Twisted python is a non-starter -- the core as we know it today, must remain clean. I think this would could be structed by bundling the 'default' blocking reactor that handles step 2, making the request, using httplib. An separate module, libcloud.async.twisted, would import all the same symbols, but change the request reactor to use Twisted's http library, instead of httplib. I think this is viable. I think drawing conclusions right now is too early, we would need to prototype it to really see if it caused more pain than the features it added. Thanks, Paul
