Hi Anthony, I used to be an active contributor to libcloud, and am technically still a committer although I don’t think I’ve been active at all since you stepped into the fold. I just wanted to start by saying thanks for your hard work!
I fully support your proposal. It will take a while to be ready to fully exploit py3, we should get started. For me what went wrong in the current codebase is that the common interfaces weren’t common enough. Even within the simple use cases there were often paper cuts (inconsistencies in exceptions raised). And the ex_ approach was off-putting too. Ultimately I ended up only targeting AWS and it was a no brainer to use botocore. An asyncio libcloud could be really compelling - but it should learn a few tricks from boto/botocore. I think there should be a low level API that targets the actual API as closely as possible. It should provide some commonality - primitives for waiting, paginating, retrying, etc. Maybe some well known exceptions. It should probably map to basic types rather than exposing XML. Fundamentally AWS APIs are inconsistent but these botocore utilities and patterns make up for it mostly. High level abstractions should be built on top of this layer and only do what is actually common between APIs - no iffy ex_ params. I wonder if we can make libcloud more attractive to providers wanting to write a python sdk? One idea I had a long time ago was to see how well the botocore data driven model could extend to other providers. I don’t really think we have the resources to do this. But the thinking was we could generate requests, asyncio and twisted bindings from the same data and potentially share the data with an node cloud driver for example. Cheers John > On 8 Mar 2018, at 23:16, anthony shaw <[email protected]> wrote: > > Hi All, > > The "cloud market" when Apache Libcloud was conceived in 2010 is very > different to how it looks today, some trends we are seeing > > 1- IaaS (our compute API) is one of many features in public clouds. Amazon, > Azure and GCP have 100's of individual services now [1]. Text-to-speech, > functions, automation, API gateways, it's impossible to keep up > 2- Private Clouds have seen a continued decline but are still popular [2] > 3- The advent of containers means it is now "easier" to deploy an > application to multiple clouds > 4- The big 3 public clouds, Amazon, Azure and Google make up most of the > cloud market [3] if you compare Apache Libcloud downloads with boto (the > native AWS Python client) downloads, it's a massive delta. Boto is in the > top 10 most popular PyPi packages > > In terms of users, I've pulled 3 snapshots of PyPi downloads, January 2016, > 2017 and 2018 [4] > Annual downloads of Apache-Libcloud have seen a slight increase, but the > 2016 - 2,778,687 > 2017 - 2,958,591 > > Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is a huge > drop. [4] > > *I would like to propose a drastic (depending on your perspective) plan to > take Apache Libcloud through to 2020* > > *1. Focus on specific use cases* > > Lack of consistency between implementations of the base class means in > practicality it's difficult to have abstracted applications. > > Apache Libcloud should (imo) come with a toolbox of utilities to both > demonstrate and validate cloud abstraction use cases, such as: > - Migrating storage objects from Cloud X to Cloud Y > - Splitting an application workload across multiple clouds > > *2. Improve performance by adopting asyncio* > > In almost all use cases, Libcloud would benefit from non-blocking calls. > Listing VM's requires multiple calls for the pages, uploading storage > objects can be done in multiple futures, deleting DNS records would be > better done in async. > > I'm suggesting we introduce a Python 3.5+ only API, move to > requests-futures or aiohttp for the base HTTP client. Yes, *I am suggesting > we drop Python 2 support in the future*. > > I've been researching how we could make this switch without breaking 64% of > our users.. > Pip now has a way to choose versions based on Python runtimes > https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4 > We could have apache-libcloud 3+ for Python 3.5 users and then maintain > 2.3 patches for Python 2.7 and 3.4 users. > > I think we can come up with a way of continuing to maintain the existing > code base for 2.7 users but move forward with a new API for async and > Python 3.5+ users. > > The downloads [4] show that Python 2.7 is still the majority of users but > this is declining quickly and by 2020 the tables will have turned. We need > to be ready for that. > > *3. Change our positioning on dependencies of 3rd party packages* > > We aren't seeing enough community contribution to keep up with the rapid > pace at which Microsoft, Amazon and Google are changing their APIs. Google > have been contributing to their driver for years. We haven't seen that from > either Amazon or Microsoft. Alibaba have contributed to theirs, many other > cloud providers have contributed, but it seems to be after APIs are > changed, not in advance. > > > * 1. Docker is another example, that API changes almost every month. The > driver we have is unstable and doesn't support API versioning correctly. * > Please consider these options, > > I would like to hear from users how you are currently using Apache Libcloud > and how you are using it > > NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly > because they represent +80% of the cloud market [3]. This is a community > developed project, has no affiliation to those vendors and these are my > opinions, not those of the project. > > [1] https://www.amazonaws.cn/en/products/ > [2] > http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises > [3] > https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240 > [4] > https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing
