Port "Using Libcloud in multi-threaded and async environments" documentation section.
Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/b6be3e70 Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/b6be3e70 Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/b6be3e70 Branch: refs/heads/trunk Commit: b6be3e70e1fe31281f537c74c9b435e52264e20c Parents: 318ed78 Author: Tomaz Muraus <[email protected]> Authored: Sat Aug 3 17:52:50 2013 +0200 Committer: Tomaz Muraus <[email protected]> Committed: Sat Aug 3 17:52:50 2013 +0200 ---------------------------------------------------------------------- docs/examples/misc/twisted_create_node.py | 34 ++++++++++ docs/index.rst | 3 +- ...-in-multithreaded-and-async-environments.rst | 69 ++++++++++++++++++++ 3 files changed, 105 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/libcloud/blob/b6be3e70/docs/examples/misc/twisted_create_node.py ---------------------------------------------------------------------- diff --git a/docs/examples/misc/twisted_create_node.py b/docs/examples/misc/twisted_create_node.py new file mode 100644 index 0000000..e0109e1 --- /dev/null +++ b/docs/examples/misc/twisted_create_node.py @@ -0,0 +1,34 @@ +from __future__ import absolute_import + +from pprint import pprint + +from twisted.internet import defer, threads, reactor +from libcloud.compute.types import Provider +from libcloud.compute.providers import get_driver + + [email protected] +def create_node(name): + node = yield threads.deferToThread(_thread_create_node, + name=name) + pprint(node) + reactor.stop() + + +def _thread_create_node(name): + Driver = get_driver(Provider.RACKSPACE) + conn = Driver('username', 'api key') + image = conn.list_images()[0] + size = conn.list_sizes()[0] + node = conn.create_node(name=name, image=image, size=size) + return node + + +def stop(*args, **kwargs): + reactor.stop() + +d = create_node(name='my-lc-node') +d.addCallback(stop) +d.addErrback(stop) + +reactor.run() http://git-wip-us.apache.org/repos/asf/libcloud/blob/b6be3e70/docs/index.rst ---------------------------------------------------------------------- diff --git a/docs/index.rst b/docs/index.rst index 09cff04..be5f382 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,14 +23,15 @@ resources: * :doc:`Load Balancers as a Service </loadbalancer/index>` * :doc:`DNS as a Service </dns/index>` - .. toctree:: :glob: + :maxdepth: 2 :hidden: compute/* storage/* loadbalancer/* dns/* + other/* .. _`Apache 2.0 license`: https://www.apache.org/licenses/LICENSE-2.0.html http://git-wip-us.apache.org/repos/asf/libcloud/blob/b6be3e70/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst ---------------------------------------------------------------------- diff --git a/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst b/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst new file mode 100644 index 0000000..26e1edb --- /dev/null +++ b/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst @@ -0,0 +1,69 @@ +Using Libcloud in multi-threaded and async environments +======================================================= + +Libcloud's primary task is to communicate with different provider APIs using +HTTP. This means most of the work is not CPU intensive, but performing all +those HTTP requests includes a lot of waiting which makes the library I/O +bound. + +Most of the time you want to perform more operations in parallel or just +want your code to finish faster (for example starting a lot of servers or +periodically polling for node status). + +Problems like this are usually solved using threads or async libraries such +as Twisted, Tornado or gevent. + +This page contains some information and tips about how to use Libcloud in +such environments. + +Libcloud and thread-safety +-------------------------- + +Important thing to keep in mind when dealing with threads is thread-safety. +Libcloud driver instance is **not** thread safe. This means if you don't want +to deal with complex (and usually inefficient) locking the easiest solution +is to create a new driver instance inside each thread. + +Using Libcloud with gevent +-------------------------- + +gevent has an ability to monkey patch and replace functions in the Python +``socket``, ``urllib2``, ``httplib`` and ``time`` module with its own +functions which don't block. + +You need to do two things when you want to use Libcloud with gevent: + +* Enable monkey patching + +.. sourcecode:: python + + from gevent import monkey + monkey.patch_all() + +* Create a separate driver instance for each Greenlet. This is necessary + because a driver instance reuses the same Connection class. + +For an example see Efficiently download multiple files using gevent. + +Using Libcloud with Twisted +--------------------------- + +Libcloud has no Twisted support included in the core which means you need +to be careful when you use it with Twisted and some other async frameworks. + +If you don't use it properly it can block the whole reactor (similar as +any other blocking library or a long CPU-intensive task) which means the +execution of other pending tasks in the event queue will be blocked. + +A simple solution to prevent blocking the reactor is to run Libcloud +calls inside a thread. In Twisted this can be achieved using +``threads.deferToThread`` which runs a provided method inside the Twisted +thread pool. + +The example bellow demonstrates how to create a new node inside a thread +without blocking the whole reactor. + +.. literalinclude:: /examples/misc/twist_create_node.py + :language: python + +
