Author: anthonyshaw
Date: Sun Apr 9 02:26:23 2017
New Revision: 1790737
URL: http://svn.apache.org/viewvc?rev=1790737&view=rev
Log:
generate website
Added:
libcloud/site/trunk/generated/blog/2017/04/09/
libcloud/site/trunk/generated/blog/2017/04/09/async-rfc.html
Modified:
libcloud/site/trunk/generated/blog/archives/2017/04/index.html
libcloud/site/trunk/generated/blog/atom.xml
libcloud/site/trunk/generated/blog/index.html
libcloud/site/trunk/generated/blog/page/10/index.html
libcloud/site/trunk/generated/blog/page/11/index.html
libcloud/site/trunk/generated/blog/page/12/index.html
libcloud/site/trunk/generated/blog/page/13/index.html
libcloud/site/trunk/generated/blog/page/14/index.html
libcloud/site/trunk/generated/blog/page/15/index.html
libcloud/site/trunk/generated/blog/page/16/index.html
libcloud/site/trunk/generated/blog/page/2/index.html
libcloud/site/trunk/generated/blog/page/3/index.html
libcloud/site/trunk/generated/blog/page/4/index.html
libcloud/site/trunk/generated/blog/page/5/index.html
libcloud/site/trunk/generated/blog/page/6/index.html
libcloud/site/trunk/generated/blog/page/7/index.html
libcloud/site/trunk/generated/blog/page/8/index.html
libcloud/site/trunk/generated/blog/page/9/index.html
libcloud/site/trunk/generated/blog/tags/news.html
libcloud/site/trunk/generated/blog/tags/tutorial.html
libcloud/site/trunk/generated/index.html
libcloud/site/trunk/generated/sitemap.xml
Added: libcloud/site/trunk/generated/blog/2017/04/09/async-rfc.html
URL:
http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/2017/04/09/async-rfc.html?rev=1790737&view=auto
==============================================================================
--- libcloud/site/trunk/generated/blog/2017/04/09/async-rfc.html (added)
+++ libcloud/site/trunk/generated/blog/2017/04/09/async-rfc.html Sun Apr 9
02:26:23 2017
@@ -0,0 +1,308 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+
+
+ <meta name="author" content="The Apache Software Foundation">
+
+
+ <meta name="keywords" content="news,tutorial" />
+
+
+
+ <title>Have your say - async support in Apache Libcloud | Apache
Libcloud</title>
+
+
+ <!-- fav icons -->
+ <link rel="shortcut icon" href="/images/favicon.png" />
+ <link rel="apple-touch-icon" href="/images/apple-touch-icon.png" />
+ <link rel="apple-touch-icon-precomposed"
href="/images/apple-touch-icon.png" />
+
+ <link href="/blog/atom.xml" type="application/atom+xml" rel="alternate"
title="Apache Libcloud Blog Feed" />
+
+ <link
href='https://fonts.googleapis.com/css?family=Open+Sans:400,400italic'
rel='stylesheet' type='text/css'>
+ <link href='https://fonts.googleapis.com/css?family=Inconsolata'
rel='stylesheet' type='text/css'>
+
+ <!-- Facebook OpenGraph tags -->
+ <meta content="Apache Libcloud" property="og:site_name">
+
+ <meta content="Have your say - async support in Apache Libcloud"
property="og:title">
+
+
+
+
+
+
+
+ <meta content="article" property="og:type">
+
+
+
+ <meta
content="https://libcloud.apache.org/blog/2017/04/09/async-rfc.html";
property="og:url">
+
+
+ <meta content="2017-04-09T00:00:00+10:00"
property="article:published_time">
+ <meta content="https://libcloud.apache.org/about.html";
property="article:author">
+
+
+
+
+ <meta content="news" property="article:tag">
+
+ <meta content="tutorial" property="article:tag">
+
+
+
+ <link href='/assets/global-eb5319c938d664847c83b2674bfd22c2.css'
rel='stylesheet' type='text/css' />
+
+ </head>
+
+ <body data-spy="scroll" data-target=".sidebar-nav" data-offset="80">
+ <nav class="navbar navbar-fixed-top navbar-inverse" role="navigation">
+ <div class="container">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse"
data-target=".navbar-ex1-collapse">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="/index.html"><img
src="/images/libcloud_logo.png" class="navbar-logo" /> Apache Libcloud</a>
+ </div>
+ <div class="collapse navbar-collapse navbar-ex1-collapse">
+ <ul class="nav navbar-nav">
+
+
+ <li ><a href="/" >Home</a></li>
+
+ <li ><a href="/about.html" >About</a></li>
+
+ <li ><a href="/getting-started.html" >Quick Start</a></li>
+
+ <li ><a href="https://libcloud.readthedocs.org/en/latest/";
target="_blank">Documentation</a></li>
+
+ <li ><a href="/downloads.html" >Downloads</a></li>
+
+ <li ><a href="/community.html" >Community</a></li>
+
+ <li ><a href="/blog/" >Blog</a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </div><!-- /.container -->
+ </nav>
+
+ <div class="container main-container">
+ <div class="row section page-content">
+ <div class="col-lg-8 col-lg-offset-2">
+
+<div class="post">
+
+ <h2>Have your say - async support in Apache Libcloud</h2>
+
+
+
+
+
+ <span class="post-date-author">By Anthony Shaw on Apr 09, 2017</span>
+
+ <div class="post-content">
+ <p>One of the big requests whilst we were replacing <code>httplib</code>
with the <code>requests</code> package in 2.0 was why didn't
+we use a HTTP library that supports <em>asynchronous</em> API calls.</p>
+
+<p>The intention for 2.0 and replacing the HTTP backend classes was to improve
the usability of the project, by making SSL
+certificates easier to manage, improving the maintainability of our source
code by using an active 3rd party package and
+also improving performance and stability.</p>
+
+<p>Apache Libcloud already has documentation on threaded libraries like gevent
and callback-based libraries like Twisted, see
+<a
href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html";>using
libcloud in multithreaded environments</a>
+for examples.</p>
+
+<p><a href="https://www.python.org/dev/peps/pep-0492/#";>PEP 492</a>,
implemented in Python 3.5 provides a new coroutine protocol using methods,
+<code>__await__</code> for classes, a coroutine method wrapper, or a method
that returns a coroutine object.
+Also async <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for";>iterators</a>
and <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with";>context
managers</a>
+have been introduced.</p>
+
+<p>We would like to take advantage of the new language features by offering
APIs in Apache Libcloud without breaking backward compatibility and
+compatibility for users of <Python 3.5.</p>
+
+<p>Use cases for this would be:</p>
+
+<ul>
+<li>Being able to fetch <code>Node</code> or <code>StorageObject</code>s from
multiple geographies or drivers simultaneously.</li>
+<li>Being able to quickly upload or download storage objects by parallelizing
operations on the <code>StorageDriver</code>.</li>
+<li>Being able to call a long-running API method (e.g. generate report),
whilst running other code.</li>
+</ul>
+
+<h2>Design 1 - async context managers <a
href="https://github.com/apache/libcloud/pull/1016";>PR 1016</a></h2>
+
+<p>This design would allow drivers to operate in 2 modes, the first is for
synchronous method calls, they return list or object
+data as per usual. The second mode, API methods like
<code>NodeDriver.list_nodes</code> would return a <a
href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects";>coroutine
object</a>
+and could be awaited or gathered using an event loop.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">import</span> <span
class="nn">asyncio</span>
+
+<span class="kn">from</span> <span class="nn">integration.driver.test</span>
<span class="kn">import</span> <span class="n">TestNodeDriver</span>
+<span class="kn">from</span> <span class="nn">libcloud.async_util</span> <span
class="kn">import</span> <span class="n">AsyncSession</span>
+
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">TestNodeDriver</span><span class="p">(</span><span
class="s">'apache'</span><span class="p">,</span> <span
class="s">'libcloud'</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="c"># regular API call</span>
+ <span class="n">nodes</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">list_nodes</span><span class="p">()</span>
+
+ <span class="n">async</span> <span class="k">with</span> <span
class="n">AsyncSession</span><span class="p">(</span><span
class="n">driver</span><span class="p">)</span> <span class="k">as</span> <span
class="n">async_instance</span><span class="p">:</span>
+ <span class="n">nodes</span> <span class="o">=</span> <span
class="n">await</span> <span class="n">async_instance</span><span
class="o">.</span><span class="n">list_nodes</span><span class="p">()</span>
+
+ <span class="k">assert</span> <span class="nb">len</span><span
class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span
class="o">==</span> <span class="mi">2</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h2>Design 2 - Additional methods in each driver for coroutines <a
href="https://github.com/apache/libcloud/pull/1027";>PR 1027</a></h2>
+
+<p>This is the second design concept for async support in Libcloud.</p>
+
+<p>The concept here is to have Asynchronous Mixins,
<code>LibcloudConnection</code> uses requests and
<code>LibcloudAsyncConnection</code> uses aiohttp for async transport <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42";>see</a></p>
+
+<p>The LibcloudAsyncConnection is an implementation detail of AsyncConnection,
which is the API for the drivers to consume <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778";>see</a></p>
+
+<p>The drivers then use this mixin for their custom connection classes,
e.g.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="k">class</span> <span
class="nc">GoogleStorageConnection</span><span class="p">(</span><span
class="n">ConnectionUserAndKey</span><span class="p">,</span> <span
class="n">AsyncConnection</span><span class="p">):</span>
+ <span class="o">...</span>
+</code></pre></div>
+<p>They then inherit from
<code>libcloud.storage.base.StorageAsyncDriver</code>, which uses a new set of
base methods, e.g. <code>iterate_containers_async</code> and can be implemented
like this:</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"> <span class="n">async</span> <span
class="k">def</span> <span class="nf">iterate_containers_async</span><span
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span
class="n">await</span> <span class="bp">self</span><span
class="o">.</span><span class="n">connection</span><span
class="o">.</span><span class="n">request_async</span><span
class="p">(</span><span class="s">'/'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">response</span><span
class="o">.</span><span class="n">status</span> <span class="o">==</span> <span
class="n">httplib</span><span class="o">.</span><span class="n">OK</span><span
class="p">:</span>
+ <span class="n">containers</span> <span class="o">=</span>
<span class="bp">self</span><span class="o">.</span><span
class="n">_to_containers</span><span class="p">(</span><span
class="n">obj</span><span class="o">=</span><span
class="n">response</span><span class="o">.</span><span
class="n">object</span><span class="p">,</span>
+ <span
class="n">xpath</span><span class="o">=</span><span
class="s">'Buckets/Bucket'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">containers</span>
+
+ <span class="k">raise</span> <span
class="n">LibcloudError</span><span class="p">(</span><span
class="s">'Unexpected status code: </span><span class="si">%s</span><span
class="s">'</span> <span class="o">%</span> <span class="p">(</span><span
class="n">response</span><span class="o">.</span><span
class="n">status</span><span class="p">),</span>
+ <span class="n">driver</span><span
class="o">=</span><span class="bp">self</span><span class="p">)</span>
+</code></pre></div>
+<p>Now the consumer can more or less do this:</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span class="kn">import</span>
<span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="kn">import</span> <span class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span
class="n">get_driver</span><span class="p">(</span><span
class="n">Provider</span><span class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">GoogleStorageDriver</span><span class="p">(</span><span
class="n">key</span><span class="o">=</span><span class="n">KEY</span><span
class="p">,</span> <span class="n">secret</span><span class="o">=</span><span
class="n">SECRET</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span
class="p">(</span><span class="n">obj</span><span class="p">):</span>
+ <span class="k">print</span><span class="p">(</span><span
class="n">obj</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="n">tasks</span> <span class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">container</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_containers_async</span><span class="p">():</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">obj</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_container_objects_async</span><span class="p">(</span><span
class="n">container</span><span class="p">):</span>
+ <span class="n">tasks</span><span class="o">.</span><span
class="n">append</span><span class="p">(</span><span
class="n">asyncio</span><span class="o">.</span><span
class="n">ensure_future</span><span class="p">(</span><span
class="n">do_stuff_with_object</span><span class="p">(</span><span
class="n">obj</span><span class="p">)))</span>
+ <span class="n">await</span> <span class="n">asyncio</span><span
class="o">.</span><span class="n">gather</span><span class="p">(</span><span
class="o">*</span><span class="n">tasks</span><span class="p">)</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h2>Design 3 - Initializer with "<em>async</em>" mode</h2>
+
+<p>This option is similar to 2, except that if a driver is instantiated with
"<code>async=True</code>",
+then all driver class methods would return coroutine objects. Internally, it
would
+patch the Connection class with the AsyncConnection class.</p>
+
+<p>The downside of this is that all method calls to a driver would need to be
awaited or used
+by an event loop.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span class="kn">import</span>
<span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="kn">import</span> <span class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span
class="n">get_driver</span><span class="p">(</span><span
class="n">Provider</span><span class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">GoogleStorageDriver</span><span class="p">(</span><span
class="n">key</span><span class="o">=</span><span class="n">KEY</span><span
class="p">,</span> <span class="n">secret</span><span class="o">=</span><span
class="n">SECRET</span><span class="p">,</span> <span
class="n">async</span><span class="o">=</span><span class="bp">True</span><span
class="p">)</span>
+
+<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span
class="p">(</span><span class="n">obj</span><span class="p">):</span>
+ <span class="k">print</span><span class="p">(</span><span
class="n">obj</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="n">tasks</span> <span class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">container</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_containers</span><span class="p">():</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">obj</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_container_objects</span><span class="p">(</span><span
class="n">container</span><span class="p">):</span>
+ <span class="n">tasks</span><span class="o">.</span><span
class="n">append</span><span class="p">(</span><span
class="n">asyncio</span><span class="o">.</span><span
class="n">ensure_future</span><span class="p">(</span><span
class="n">do_stuff_with_object</span><span class="p">(</span><span
class="n">obj</span><span class="p">)))</span>
+ <span class="n">await</span> <span class="n">asyncio</span><span
class="o">.</span><span class="n">gather</span><span class="p">(</span><span
class="o">*</span><span class="n">tasks</span><span class="p">)</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h1>Give us feedback</h1>
+
+<p>Got a better idea? Have an API or design, the question we're asking is
+"if you wanted to use Libcloud for an async application, what would the
code look like?" This helps us design
+the API and the implementation details can follow.</p>
+
+<p>Feel free to comment on the mailing list or on the pull requests, or raise
your own pull-request with an API design.</p>
+
+ </div>
+
+ <div class="row section post-meta">
+ <div class="col-md-12 post-tags">
+ <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a
href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p>
+ </div>
+ </div>
+</div>
+
+Share : <a href="https://twitter.com/share";
class="twitter-share-button"{count} data-via="libcloud" data-size="large"
data-hashtags="cloud">Tweet</a>
+<script>!function(d,s,id){var
js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document,
'script', 'twitter-wjs');</script>
+
+ </div>
+</div>
+
+
+ <hr />
+
+ <footer>
+ <div class="row">
+ <div class="col-lg-12 text-center">
+ <div class="footer-links">
+ <p><a href="http://www.apache.org/licenses/";>License</a> | <a
+ href="/security.html">Security</a> | <a
+ href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a> |
+ <a href="http://www.apache.org/foundation/thanks.html";>Thanks</a> |
+ <a href="/credits.html">Credits</a> | <a href="/media.html">Media</a>
+</div>
+
+<div class="footer-text">
+ <p class="">Copyright © 2009-2017 <a href="https://www.apache.org/";
target="_blank">The Apache Software Foundation</a></p>
+ <p class="">Apache Libcloud, Libcloud, Apache, the Apache feather, and the
Apache Libcloud project logo are trademarks of the Apache Software Foundation.
All other marks mentioned may be trademarks or registered trademarks of their
respective owners.</p>
+</div>
+
+ </div>
+ </div>
+ </footer>
+
+ </div><!-- /.container -->
+
+ <!-- JavaScript -->
+ <script src='/assets/global-c6949aebbeb9952ae7967e9bc1aedfbc.js'
type='text/javascript'></script>
+
+
+
+
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-23580482-1']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript';
ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' :
'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
+ })();
+</script>
+
+ </body>
+</html>
Modified: libcloud/site/trunk/generated/blog/archives/2017/04/index.html
URL:
http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/archives/2017/04/index.html?rev=1790737&r1=1790736&r2=1790737&view=diff
==============================================================================
--- libcloud/site/trunk/generated/blog/archives/2017/04/index.html (original)
+++ libcloud/site/trunk/generated/blog/archives/2017/04/index.html Sun Apr 9
02:26:23 2017
@@ -95,6 +95,163 @@
<div class="post">
+ <h2><a href="/blog/2017/04/09/async-rfc.html">Have your say - async
support in Apache Libcloud</a></h2>
+
+
+
+
+
+ <span class="post-date-author">By Anthony Shaw on Apr 09, 2017</span>
+
+ <div class="post-content">
+ <p>One of the big requests whilst we were replacing <code>httplib</code>
with the <code>requests</code> package in 2.0 was why didn't
+we use a HTTP library that supports <em>asynchronous</em> API calls.</p>
+
+<p>The intention for 2.0 and replacing the HTTP backend classes was to improve
the usability of the project, by making SSL
+certificates easier to manage, improving the maintainability of our source
code by using an active 3rd party package and
+also improving performance and stability.</p>
+
+<p>Apache Libcloud already has documentation on threaded libraries like gevent
and callback-based libraries like Twisted, see
+<a
href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html";>using
libcloud in multithreaded environments</a>
+for examples.</p>
+
+<p><a href="https://www.python.org/dev/peps/pep-0492/#";>PEP 492</a>,
implemented in Python 3.5 provides a new coroutine protocol using methods,
+<code>__await__</code> for classes, a coroutine method wrapper, or a method
that returns a coroutine object.
+Also async <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for";>iterators</a>
and <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with";>context
managers</a>
+have been introduced.</p>
+
+<p>We would like to take advantage of the new language features by offering
APIs in Apache Libcloud without breaking backward compatibility and
+compatibility for users of <Python 3.5.</p>
+
+<p>Use cases for this would be:</p>
+
+<ul>
+<li>Being able to fetch <code>Node</code> or <code>StorageObject</code>s from
multiple geographies or drivers simultaneously.</li>
+<li>Being able to quickly upload or download storage objects by parallelizing
operations on the <code>StorageDriver</code>.</li>
+<li>Being able to call a long-running API method (e.g. generate report),
whilst running other code.</li>
+</ul>
+
+<h2>Design 1 - async context managers <a
href="https://github.com/apache/libcloud/pull/1016";>PR 1016</a></h2>
+
+<p>This design would allow drivers to operate in 2 modes, the first is for
synchronous method calls, they return list or object
+data as per usual. The second mode, API methods like
<code>NodeDriver.list_nodes</code> would return a <a
href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects";>coroutine
object</a>
+and could be awaited or gathered using an event loop.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">import</span> <span
class="nn">asyncio</span>
+
+<span class="kn">from</span> <span class="nn">integration.driver.test</span>
<span class="kn">import</span> <span class="n">TestNodeDriver</span>
+<span class="kn">from</span> <span class="nn">libcloud.async_util</span> <span
class="kn">import</span> <span class="n">AsyncSession</span>
+
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">TestNodeDriver</span><span class="p">(</span><span
class="s">'apache'</span><span class="p">,</span> <span
class="s">'libcloud'</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="c"># regular API call</span>
+ <span class="n">nodes</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">list_nodes</span><span class="p">()</span>
+
+ <span class="n">async</span> <span class="k">with</span> <span
class="n">AsyncSession</span><span class="p">(</span><span
class="n">driver</span><span class="p">)</span> <span class="k">as</span> <span
class="n">async_instance</span><span class="p">:</span>
+ <span class="n">nodes</span> <span class="o">=</span> <span
class="n">await</span> <span class="n">async_instance</span><span
class="o">.</span><span class="n">list_nodes</span><span class="p">()</span>
+
+ <span class="k">assert</span> <span class="nb">len</span><span
class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span
class="o">==</span> <span class="mi">2</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h2>Design 2 - Additional methods in each driver for coroutines <a
href="https://github.com/apache/libcloud/pull/1027";>PR 1027</a></h2>
+
+<p>This is the second design concept for async support in Libcloud.</p>
+
+<p>The concept here is to have Asynchronous Mixins,
<code>LibcloudConnection</code> uses requests and
<code>LibcloudAsyncConnection</code> uses aiohttp for async transport <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42";>see</a></p>
+
+<p>The LibcloudAsyncConnection is an implementation detail of AsyncConnection,
which is the API for the drivers to consume <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778";>see</a></p>
+
+<p>The drivers then use this mixin for their custom connection classes,
e.g.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="k">class</span> <span
class="nc">GoogleStorageConnection</span><span class="p">(</span><span
class="n">ConnectionUserAndKey</span><span class="p">,</span> <span
class="n">AsyncConnection</span><span class="p">):</span>
+ <span class="o">...</span>
+</code></pre></div>
+<p>They then inherit from
<code>libcloud.storage.base.StorageAsyncDriver</code>, which uses a new set of
base methods, e.g. <code>iterate_containers_async</code> and can be implemented
like this:</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"> <span class="n">async</span> <span
class="k">def</span> <span class="nf">iterate_containers_async</span><span
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span
class="n">await</span> <span class="bp">self</span><span
class="o">.</span><span class="n">connection</span><span
class="o">.</span><span class="n">request_async</span><span
class="p">(</span><span class="s">'/'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">response</span><span
class="o">.</span><span class="n">status</span> <span class="o">==</span> <span
class="n">httplib</span><span class="o">.</span><span class="n">OK</span><span
class="p">:</span>
+ <span class="n">containers</span> <span class="o">=</span>
<span class="bp">self</span><span class="o">.</span><span
class="n">_to_containers</span><span class="p">(</span><span
class="n">obj</span><span class="o">=</span><span
class="n">response</span><span class="o">.</span><span
class="n">object</span><span class="p">,</span>
+ <span
class="n">xpath</span><span class="o">=</span><span
class="s">'Buckets/Bucket'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">containers</span>
+
+ <span class="k">raise</span> <span
class="n">LibcloudError</span><span class="p">(</span><span
class="s">'Unexpected status code: </span><span class="si">%s</span><span
class="s">'</span> <span class="o">%</span> <span class="p">(</span><span
class="n">response</span><span class="o">.</span><span
class="n">status</span><span class="p">),</span>
+ <span class="n">driver</span><span
class="o">=</span><span class="bp">self</span><span class="p">)</span>
+</code></pre></div>
+<p>Now the consumer can more or less do this:</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span class="kn">import</span>
<span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="kn">import</span> <span class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span
class="n">get_driver</span><span class="p">(</span><span
class="n">Provider</span><span class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">GoogleStorageDriver</span><span class="p">(</span><span
class="n">key</span><span class="o">=</span><span class="n">KEY</span><span
class="p">,</span> <span class="n">secret</span><span class="o">=</span><span
class="n">SECRET</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span
class="p">(</span><span class="n">obj</span><span class="p">):</span>
+ <span class="k">print</span><span class="p">(</span><span
class="n">obj</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="n">tasks</span> <span class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">container</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_containers_async</span><span class="p">():</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">obj</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_container_objects_async</span><span class="p">(</span><span
class="n">container</span><span class="p">):</span>
+ <span class="n">tasks</span><span class="o">.</span><span
class="n">append</span><span class="p">(</span><span
class="n">asyncio</span><span class="o">.</span><span
class="n">ensure_future</span><span class="p">(</span><span
class="n">do_stuff_with_object</span><span class="p">(</span><span
class="n">obj</span><span class="p">)))</span>
+ <span class="n">await</span> <span class="n">asyncio</span><span
class="o">.</span><span class="n">gather</span><span class="p">(</span><span
class="o">*</span><span class="n">tasks</span><span class="p">)</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h2>Design 3 - Initializer with "<em>async</em>" mode</h2>
+
+<p>This option is similar to 2, except that if a driver is instantiated with
"<code>async=True</code>",
+then all driver class methods would return coroutine objects. Internally, it
would
+patch the Connection class with the AsyncConnection class.</p>
+
+<p>The downside of this is that all method calls to a driver would need to be
awaited or used
+by an event loop.</p>
+<div class="highlight"><pre><code class="python language-python"
data-lang="python"><span class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span class="kn">import</span>
<span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="kn">import</span> <span class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span class="o">=</span> <span
class="n">get_driver</span><span class="p">(</span><span
class="n">Provider</span><span class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span
class="n">GoogleStorageDriver</span><span class="p">(</span><span
class="n">key</span><span class="o">=</span><span class="n">KEY</span><span
class="p">,</span> <span class="n">secret</span><span class="o">=</span><span
class="n">SECRET</span><span class="p">,</span> <span
class="n">async</span><span class="o">=</span><span class="bp">True</span><span
class="p">)</span>
+
+<span class="k">def</span> <span class="nf">do_stuff_with_object</span><span
class="p">(</span><span class="n">obj</span><span class="p">):</span>
+ <span class="k">print</span><span class="p">(</span><span
class="n">obj</span><span class="p">)</span>
+
+<span class="n">async</span> <span class="k">def</span> <span
class="nf">run</span><span class="p">():</span>
+ <span class="n">tasks</span> <span class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">container</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_containers</span><span class="p">():</span>
+ <span class="n">async</span> <span class="k">for</span> <span
class="n">obj</span> <span class="ow">in</span> <span
class="n">driver</span><span class="o">.</span><span
class="n">iterate_container_objects</span><span class="p">(</span><span
class="n">container</span><span class="p">):</span>
+ <span class="n">tasks</span><span class="o">.</span><span
class="n">append</span><span class="p">(</span><span
class="n">asyncio</span><span class="o">.</span><span
class="n">ensure_future</span><span class="p">(</span><span
class="n">do_stuff_with_object</span><span class="p">(</span><span
class="n">obj</span><span class="p">)))</span>
+ <span class="n">await</span> <span class="n">asyncio</span><span
class="o">.</span><span class="n">gather</span><span class="p">(</span><span
class="o">*</span><span class="n">tasks</span><span class="p">)</span>
+
+<span class="n">loop</span> <span class="o">=</span> <span
class="n">asyncio</span><span class="o">.</span><span
class="n">get_event_loop</span><span class="p">()</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">run_until_complete</span><span class="p">(</span><span
class="n">run</span><span class="p">())</span>
+<span class="n">loop</span><span class="o">.</span><span
class="n">close</span><span class="p">()</span>
+</code></pre></div>
+<h1>Give us feedback</h1>
+
+<p>Got a better idea? Have an API or design, the question we're asking is
+"if you wanted to use Libcloud for an async application, what would the
code look like?" This helps us design
+the API and the implementation details can follow.</p>
+
+<p>Feel free to comment on the mailing list or on the pull requests, or raise
your own pull-request with an API design.</p>
+
+ </div>
+
+ <div class="row section post-meta">
+ <div class="col-md-12 post-tags">
+ <p>Tags: <a href="/blog/tags/news.html" rel="tag">news</a>, <a
href="/blog/tags/tutorial.html" rel="tag">tutorial</a></p>
+ </div>
+ </div>
+</div>
+
+
+ <div class="post">
+
<h2><a href="/blog/2017/04/07/libcloud-2-0-0-rc2-released.html">Libcloud
2.0.0rc2 released</a></h2>
Modified: libcloud/site/trunk/generated/blog/atom.xml
URL:
http://svn.apache.org/viewvc/libcloud/site/trunk/generated/blog/atom.xml?rev=1790737&r1=1790736&r2=1790737&view=diff
==============================================================================
--- libcloud/site/trunk/generated/blog/atom.xml (original)
+++ libcloud/site/trunk/generated/blog/atom.xml Sun Apr 9 02:26:23 2017
@@ -5,13 +5,159 @@
<link href="https://libcloud.apache.org"; rel="self" />
<link href="https://libcloud.apache.org"; />
<id>https://libcloud.apache.org</id>
- <updated>2017-04-08T12:08:14+10:00</updated>
+ <updated>2017-04-09T12:25:24+10:00</updated>
<author>
<name>The Apache Software Foundation</name>
</author>
<entry>
+ <title>Have your say - async support in Apache Libcloud</title>
+ <link href="https://libcloud.apache.org/blog/2017/04/09/async-rfc.html"; />
+ <id>https://libcloud.apache.org/blog/2017/04/09/async-rfc.html</id>
+ <updated>2017-04-09T00:00:00+10:00</updated>
+ <author>
+ <name>Anthony Shaw</name>
+ </author>
+ <content type="html"><p>One of the big requests whilst we were
replacing <code>httplib</code> with the
<code>requests</code> package in 2.0 was why didn&#39;t
+we use a HTTP library that supports <em>asynchronous</em> API
calls.</p>
+
+<p>The intention for 2.0 and replacing the HTTP backend classes was to
improve the usability of the project, by making SSL
+certificates easier to manage, improving the maintainability of our source
code by using an active 3rd party package and
+also improving performance and stability.</p>
+
+<p>Apache Libcloud already has documentation on threaded libraries like
gevent and callback-based libraries like Twisted, see
+<a
href="https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html">using
libcloud in multithreaded environments</a>
+for examples.</p>
+
+<p><a
href="https://www.python.org/dev/peps/pep-0492/#">PEP
492</a>, implemented in Python 3.5 provides a new coroutine protocol
using methods,
+<code>__await__</code> for classes, a coroutine method wrapper, or
a method that returns a coroutine object.
+Also async <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for">iterators</a>
and <a
href="https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with">context
managers</a>
+have been introduced.</p>
+
+<p>We would like to take advantage of the new language features by
offering APIs in Apache Libcloud without breaking backward compatibility and
+compatibility for users of &lt;Python 3.5.</p>
+
+<p>Use cases for this would be:</p>
+
+<ul>
+<li>Being able to fetch <code>Node</code> or
<code>StorageObject</code>s from multiple geographies or drivers
simultaneously.</li>
+<li>Being able to quickly upload or download storage objects by
parallelizing operations on the
<code>StorageDriver</code>.</li>
+<li>Being able to call a long-running API method (e.g. generate report),
whilst running other code.</li>
+</ul>
+
+<h2>Design 1 - async context managers <a
href="https://github.com/apache/libcloud/pull/1016">PR
1016</a></h2>
+
+<p>This design would allow drivers to operate in 2 modes, the first is
for synchronous method calls, they return list or object
+data as per usual. The second mode, API methods like
<code>NodeDriver.list_nodes</code> would return a <a
href="https://www.python.org/dev/peps/pep-0492/#coroutine-objects">coroutine
object</a>
+and could be awaited or gathered using an event loop.</p>
+<div class="highlight"><pre><code class="python
language-python" data-lang="python"><span
class="kn">import</span> <span
class="nn">asyncio</span>
+
+<span class="kn">from</span> <span
class="nn">integration.driver.test</span> <span
class="kn">import</span> <span
class="n">TestNodeDriver</span>
+<span class="kn">from</span> <span
class="nn">libcloud.async_util</span> <span
class="kn">import</span> <span
class="n">AsyncSession</span>
+
+<span class="n">driver</span> <span
class="o">=</span> <span
class="n">TestNodeDriver</span><span
class="p">(</span><span
class="s">&#39;apache&#39;</span><span
class="p">,</span> <span
class="s">&#39;libcloud&#39;</span><span
class="p">)</span>
+
+<span class="n">async</span> <span
class="k">def</span> <span
class="nf">run</span><span
class="p">():</span>
+ <span class="c"># regular API call</span>
+ <span class="n">nodes</span> <span
class="o">=</span> <span
class="n">driver</span><span
class="o">.</span><span
class="n">list_nodes</span><span
class="p">()</span>
+
+ <span class="n">async</span> <span
class="k">with</span> <span
class="n">AsyncSession</span><span
class="p">(</span><span
class="n">driver</span><span
class="p">)</span> <span
class="k">as</span> <span
class="n">async_instance</span><span
class="p">:</span>
+ <span class="n">nodes</span> <span
class="o">=</span> <span
class="n">await</span> <span
class="n">async_instance</span><span
class="o">.</span><span
class="n">list_nodes</span><span
class="p">()</span>
+
+ <span class="k">assert</span> <span
class="nb">len</span><span
class="p">(</span><span
class="n">nodes</span><span
class="p">)</span> <span
class="o">==</span> <span
class="mi">2</span>
+
+<span class="n">loop</span> <span
class="o">=</span> <span
class="n">asyncio</span><span
class="o">.</span><span
class="n">get_event_loop</span><span
class="p">()</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">run_until_complete</span><span
class="p">(</span><span
class="n">run</span><span
class="p">())</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">close</span><span
class="p">()</span>
+</code></pre></div>
+<h2>Design 2 - Additional methods in each driver for coroutines <a
href="https://github.com/apache/libcloud/pull/1027">PR
1027</a></h2>
+
+<p>This is the second design concept for async support in
Libcloud.</p>
+
+<p>The concept here is to have Asynchronous Mixins,
<code>LibcloudConnection</code> uses requests and
<code>LibcloudAsyncConnection</code> uses aiohttp for async
transport <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42">see</a></p>
+
+<p>The LibcloudAsyncConnection is an implementation detail of
AsyncConnection, which is the API for the drivers to consume <a
href="https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778">see</a></p>
+
+<p>The drivers then use this mixin for their custom connection classes,
e.g.</p>
+<div class="highlight"><pre><code class="python
language-python" data-lang="python"><span
class="k">class</span> <span
class="nc">GoogleStorageConnection</span><span
class="p">(</span><span
class="n">ConnectionUserAndKey</span><span
class="p">,</span> <span
class="n">AsyncConnection</span><span
class="p">):</span>
+ <span class="o">...</span>
+</code></pre></div>
+<p>They then inherit from
<code>libcloud.storage.base.StorageAsyncDriver</code>, which uses a
new set of base methods, e.g. <code>iterate_containers_async</code>
and can be implemented like this:</p>
+<div class="highlight"><pre><code class="python
language-python" data-lang="python"> <span
class="n">async</span> <span
class="k">def</span> <span
class="nf">iterate_containers_async</span><span
class="p">(</span><span
class="bp">self</span><span
class="p">):</span>
+ <span class="n">response</span> <span
class="o">=</span> <span
class="n">await</span> <span
class="bp">self</span><span
class="o">.</span><span
class="n">connection</span><span
class="o">.</span><span
class="n">request_async</span><span
class="p">(</span><span
class="s">&#39;/&#39;</span><span
class="p">)</span>
+ <span class="k">if</span> <span
class="n">response</span><span
class="o">.</span><span
class="n">status</span> <span
class="o">==</span> <span
class="n">httplib</span><span
class="o">.</span><span
class="n">OK</span><span
class="p">:</span>
+ <span class="n">containers</span>
<span class="o">=</span> <span
class="bp">self</span><span
class="o">.</span><span
class="n">_to_containers</span><span
class="p">(</span><span
class="n">obj</span><span
class="o">=</span><span
class="n">response</span><span
class="o">.</span><span
class="n">object</span><span
class="p">,</span>
+ <span
class="n">xpath</span><span
class="o">=</span><span
class="s">&#39;Buckets/Bucket&#39;</span><span
class="p">)</span>
+ <span class="k">return</span> <span
class="n">containers</span>
+
+ <span class="k">raise</span> <span
class="n">LibcloudError</span><span
class="p">(</span><span
class="s">&#39;Unexpected status code: </span><span
class="si">%s</span><span
class="s">&#39;</span> <span
class="o">%</span> <span
class="p">(</span><span
class="n">response</span><span
class="o">.</span><span
class="n">status</span><span
class="p">),</span>
+ <span
class="n">driver</span><span
class="o">=</span><span
class="bp">self</span><span
class="p">)</span>
+</code></pre></div>
+<p>Now the consumer can more or less do this:</p>
+<div class="highlight"><pre><code class="python
language-python" data-lang="python"><span
class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span
class="kn">import</span> <span
class="n">get_driver</span>
+<span class="kn">from</span> <span
class="nn">libcloud.storage.types</span> <span
class="kn">import</span> <span
class="n">Provider</span>
+
+<span class="kn">import</span> <span
class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span
class="o">=</span> <span
class="n">get_driver</span><span
class="p">(</span><span
class="n">Provider</span><span
class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span
class="p">)</span>
+<span class="n">driver</span> <span
class="o">=</span> <span
class="n">GoogleStorageDriver</span><span
class="p">(</span><span
class="n">key</span><span
class="o">=</span><span
class="n">KEY</span><span
class="p">,</span> <span
class="n">secret</span><span
class="o">=</span><span
class="n">SECRET</span><span
class="p">)</span>
+
+<span class="k">def</span> <span
class="nf">do_stuff_with_object</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">):</span>
+ <span class="k">print</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">)</span>
+
+<span class="n">async</span> <span
class="k">def</span> <span
class="nf">run</span><span
class="p">():</span>
+ <span class="n">tasks</span> <span
class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span
class="k">for</span> <span
class="n">container</span> <span
class="ow">in</span> <span
class="n">driver</span><span
class="o">.</span><span
class="n">iterate_containers_async</span><span
class="p">():</span>
+ <span class="n">async</span> <span
class="k">for</span> <span
class="n">obj</span> <span
class="ow">in</span> <span
class="n">driver</span><span
class="o">.</span><span
class="n">iterate_container_objects_async</span><span
class="p">(</span><span
class="n">container</span><span
class="p">):</span>
+ <span class="n">tasks</span><span
class="o">.</span><span
class="n">append</span><span
class="p">(</span><span
class="n">asyncio</span><span
class="o">.</span><span
class="n">ensure_future</span><span
class="p">(</span><span
class="n">do_stuff_with_object</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">)))</span>
+ <span class="n">await</span> <span
class="n">asyncio</span><span
class="o">.</span><span
class="n">gather</span><span
class="p">(</span><span
class="o">*</span><span
class="n">tasks</span><span
class="p">)</span>
+
+<span class="n">loop</span> <span
class="o">=</span> <span
class="n">asyncio</span><span
class="o">.</span><span
class="n">get_event_loop</span><span
class="p">()</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">run_until_complete</span><span
class="p">(</span><span
class="n">run</span><span
class="p">())</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">close</span><span
class="p">()</span>
+</code></pre></div>
+<h2>Design 3 - Initializer with
&quot;<em>async</em>&quot; mode</h2>
+
+<p>This option is similar to 2, except that if a driver is instantiated
with &quot;<code>async=True</code>&quot;,
+then all driver class methods would return coroutine objects. Internally, it
would
+patch the Connection class with the AsyncConnection class.</p>
+
+<p>The downside of this is that all method calls to a driver would need
to be awaited or used
+by an event loop.</p>
+<div class="highlight"><pre><code class="python
language-python" data-lang="python"><span
class="kn">from</span> <span
class="nn">libcloud.storage.providers</span> <span
class="kn">import</span> <span
class="n">get_driver</span>
+<span class="kn">from</span> <span
class="nn">libcloud.storage.types</span> <span
class="kn">import</span> <span
class="n">Provider</span>
+
+<span class="kn">import</span> <span
class="nn">asyncio</span>
+
+<span class="n">GoogleStorageDriver</span> <span
class="o">=</span> <span
class="n">get_driver</span><span
class="p">(</span><span
class="n">Provider</span><span
class="o">.</span><span
class="n">GOOGLE_STORAGE</span><span
class="p">)</span>
+<span class="n">driver</span> <span
class="o">=</span> <span
class="n">GoogleStorageDriver</span><span
class="p">(</span><span
class="n">key</span><span
class="o">=</span><span
class="n">KEY</span><span
class="p">,</span> <span
class="n">secret</span><span
class="o">=</span><span
class="n">SECRET</span><span
class="p">,</span> <span
class="n">async</span><span
class="o">=</span><span
class="bp">True</span><span
class="p">)</span>
+
+<span class="k">def</span> <span
class="nf">do_stuff_with_object</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">):</span>
+ <span class="k">print</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">)</span>
+
+<span class="n">async</span> <span
class="k">def</span> <span
class="nf">run</span><span
class="p">():</span>
+ <span class="n">tasks</span> <span
class="o">=</span> <span
class="p">[]</span>
+ <span class="n">async</span> <span
class="k">for</span> <span
class="n">container</span> <span
class="ow">in</span> <span
class="n">driver</span><span
class="o">.</span><span
class="n">iterate_containers</span><span
class="p">():</span>
+ <span class="n">async</span> <span
class="k">for</span> <span
class="n">obj</span> <span
class="ow">in</span> <span
class="n">driver</span><span
class="o">.</span><span
class="n">iterate_container_objects</span><span
class="p">(</span><span
class="n">container</span><span
class="p">):</span>
+ <span class="n">tasks</span><span
class="o">.</span><span
class="n">append</span><span
class="p">(</span><span
class="n">asyncio</span><span
class="o">.</span><span
class="n">ensure_future</span><span
class="p">(</span><span
class="n">do_stuff_with_object</span><span
class="p">(</span><span
class="n">obj</span><span
class="p">)))</span>
+ <span class="n">await</span> <span
class="n">asyncio</span><span
class="o">.</span><span
class="n">gather</span><span
class="p">(</span><span
class="o">*</span><span
class="n">tasks</span><span
class="p">)</span>
+
+<span class="n">loop</span> <span
class="o">=</span> <span
class="n">asyncio</span><span
class="o">.</span><span
class="n">get_event_loop</span><span
class="p">()</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">run_until_complete</span><span
class="p">(</span><span
class="n">run</span><span
class="p">())</span>
+<span class="n">loop</span><span
class="o">.</span><span
class="n">close</span><span
class="p">()</span>
+</code></pre></div>
+<h1>Give us feedback</h1>
+
+<p>Got a better idea? Have an API or design, the question we&#39;re
asking is
+&quot;if you wanted to use Libcloud for an async application, what would
the code look like?&quot; This helps us design
+the API and the implementation details can follow.</p>
+
+<p>Feel free to comment on the mailing list or on the pull requests, or
raise your own pull-request with an API design.</p>
+</content>
+ </entry>
+
+ <entry>
<title>Libcloud 2.0.0rc2 released</title>
<link
href="https://libcloud.apache.org/blog/2017/04/07/libcloud-2-0-0-rc2-released.html";
/>
<id>https://libcloud.apache.org/blog/2017/04/07/libcloud-2-0-0-rc2-released.html</id>
@@ -1239,75 +1385,4 @@ that there are no regression bugs with t
</content>
</entry>
- <entry>
- <title>New compute drivers and deprecated drivers in 1.0</title>
- <link
href="https://libcloud.apache.org/blog/2016/02/16/new-drivers-deprecated-drivers.html";
/>
-
<id>https://libcloud.apache.org/blog/2016/02/16/new-drivers-deprecated-drivers.html</id>
- <updated>2016-02-16T00:00:00+11:00</updated>
- <author>
- <name>Anthony Shaw</name>
- </author>
- <content type="html"><p>With Libcloud 1.0.0 around the corner,
it&#39;s time to have a spring clean of the compute drivers. Granted,
it&#39;s not spring everywhere -actually
-I&#39;m writing from Sydney, Australia where it&#39;s definitely
summer.</p>
-
-<p>Looking at the 52 providers in the 0.21.0 release, I have identified
5 providers that are no longer available or open.</p>
-
-<ul>
-<li>CloudFrames
-
-<ul>
-<li>Looks dead - website doesn&#39;t work, can&#39;t see any
references to this online.</li>
-<li><code>libcloud.compute.drivers.cloudframes</code></li>
-</ul></li>
-<li>HP Public Cloud (Helion)
-
-<ul>
-<li><a
href="https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2566">Shut
down on 31st of January 2016</a></li>
-<li><code>libcloud.compute.drivers.hpcloud</code></li>
-</ul></li>
-<li>IBM SmartCloud Enterprise
-
-<ul>
-<li><a
href="http://www.theregister.co.uk/2013/10/31/ibm_to_shutter_smartcloud_enterprise_moves_customers_to_softlayer/">Shut
down 2014, customers instructed to move to SoftLayer</a></li>
-<li><code>libcloud.compute.drivers.ibm_sce</code></li>
-</ul></li>
-<li>Ninefold
-
-<ul>
-<li><a
href="http://www.itnews.com.au/news/ninefold-to-shut-down-411312">Ninefold
shut down in January 2016</a></li>
-<li><code>libcloud.compute.drivers.ninefold</code></li>
-</ul></li>
-<li>Opsource
-
-<ul>
-<li>Dimension Data acquired OpSource in 2012, the Opsource driver is
succeeded by the Dimension Data driver.</li>
-<li><code>libcloud.compute.drivers.opsource</code></li>
-</ul></li>
-</ul>
-
-<h3>Handling deprecated drivers</h3>
-
-<p>For 1.0.0, we need a clean and user-friendly way of handling
deprecated drivers as well as keeping the repository clean from legacy
code.</p>
-
-<p>The most obvious implementation is that calls to
<code>get_driver(Provider.NINEFOLD)</code> as an example will
return a user error message saying
-this provider is no longer supported with a link to a new article and an
alternative solution.</p>
-
-<p>Currently, users trying to instantiate a HPE public cloud driver for
example will get a connection error, which is not user friendly.</p>
-
-<h3>New compute drivers in 1.0.0-pre2</h3>
-
-<p>The upcoming release, so currently available in trunk contains some
new compute drivers.</p>
-
-<ul>
-<li><a
href="http://libcloud.readthedocs.org/en/latest/compute/drivers/ntta.html">NTT
America Public Cloud</a></li>
-<li><a
href="http://libcloud.readthedocs.org/en/latest/compute/drivers/internetsolutions.html">Internet
Solutions Public Cloud</a></li>
-<li><a
href="http://libcloud.readthedocs.org/en/latest/compute/drivers/medone.html">Med-1
Public Cloud</a></li>
-<li><a
href="http://libcloud.readthedocs.org/en/latest/compute/drivers/indosat.html">Indosat
Cloud</a></li>
-<li><a
href="http://libcloud.readthedocs.org/en/latest/compute/drivers/bsnl.html">BSNL
IDC Cloud</a></li>
-</ul>
-
-<p>Full change log can be found at <a
href="https://github.com/apache/libcloud/blob/trunk/CHANGES.rst">here</a>.</p>
-</content>
- </entry>
-
</feed>
