Re: [openstack-dev] Proposal for the new service: MagnetoDB, DynamoDB API implementation for OpenStack

[Ilya Sviridov]

Hi Thierry,

Yes, you are right. AWS SimpleDB and DynamoDB look alike.
I think in API implementation itself there are a lot of common parts.

But exactly backend makes the difference.

Planning to engage Trove for datasource provisioning and management, the
key features of MagnetoDB is autoscaling and high-availability.
By design we plan to achieve the constant response time regardless of
amount of data.
In order to achieve such qualities the backend part should satisfy that
requirement itself and obviously not always can be applicable for general
cases.

However I believe that Soren have made huge progress, of course I would
like to discus it and maybe join our efforts.

I wrote him without response yet.

Thank you for input.

With best regards,
Ilya Sviridov




On Thu, Oct 31, 2013 at 12:01 PM, Thierry Carrez wrote:

> Ilya Sviridov wrote:
> > I would like to propose a new initiative to implement AWS DynamoDB API
> > for OpenStack.
> > [...]
>
> There is a parallel effort by Soren to create an equivalent of SimpleDB:
> http://blog.linux2go.dk/tag/basicdb/
>
> I'm not familiar enough with SimpleDB/DynamoDB, but I wonder if we
> shouldn't have a single service to handle both cases, as the backend
> side looks reusable to me.
>
> You guys might want to talk together.
>
> --
> Thierry Carrez (ttx)
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal for the new service: MagnetoDB, DynamoDB API implementation for OpenStack

[Thierry Carrez]

Ilya Sviridov wrote:
> I would like to propose a new initiative to implement AWS DynamoDB API
> for OpenStack.
> [...]

There is a parallel effort by Soren to create an equivalent of SimpleDB:
http://blog.linux2go.dk/tag/basicdb/

I'm not familiar enough with SimpleDB/DynamoDB, but I wonder if we
shouldn't have a single service to handle both cases, as the backend
side looks reusable to me.

You guys might want to talk together.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Proposal for the new service: MagnetoDB, DynamoDB API implementation for OpenStack

[Ilya Sviridov]

Hi, Openstackers

I would like to propose a new initiative to implement AWS DynamoDB API for
OpenStack.

I believe this is a right moment for OpenStack community to start MagnetoDB
discussion and design around such service as more data processing and
management services, such as Savanna and Trove, are appearing in OpenStack,
creating a sufficient basis for higher level data APIs like DynamoDB API.
With Trove you can provision and manage different databases including NoSQL
databases support, which will appear soon. It minimizes the maintenance
process and turns database deployment and management to API calls.

DynamoDB [1]  is well-known key-value
NoSQL storage provided by Amazon with HTTP interface. It uses JSON as its
transfer data model. The data is stored as key-value, where value is list
of attributes. The attribute’s value can be String, Number, Binary or sets
of those types.  Read consistency can be strong or eventual per request.
The secondary indexes are also supported.

Simplicity, reliability, and pleasant documentation have made it actively
used in a lot of applications, but hosting them on OpenStack is impossible
due to the absence of such storage. DynamoDB is already widely used and has
its own niche [2]  despite
being closed and proprietary.

Let me share the idea to implement a similar service which imitates
DynamoDB API 
[3],
called MagnetoDB.

MagnetoDB will fully implement DynamoDB API thus enabling the possibility
for cloud users to migrate their applications that use DynamoDB from AWS to
OpenStack.

For backend storage provisioning and management, OpenStack DBaaS Trove will
be used.

The backend database should be pluggable to provide flexibility in terms
choosing the solution which best matched the existing or planned OpenStack
installation technology stack. MySQL is one of the obvious options, being
the de-facto standard for use with OpenStack and being supported by Trove
right away.

However, users of DynamoDB expect high performance, scalability, and
availability. Those can be achieved more easily  by using one of the NoSQL
solutions in MagnetoDB.

Apache Cassandra [4] looks very suitable for
that case, due to its tunable consistency, easy scalability, key-value data
model, ring topology and other features that give us predictable high
performance and fault tolerance. The Cassandra data model perfectly fits
MagnetoDB needs.

Moreover, support of NoSQL databases is in Trove roadmap
[5]and
the fist version of Cassandra provisioning is currently in progress
[6] .

AWS SDK 
[7]can
be used to query data. Provided by Amazon, it hides all protocol stuff
giving the high level API.  In Python world you can use the rather common
Boto [8] 
[9]library.

An  initial draft of the more official proposal is available here:
https://wiki.openstack.org/wiki/MagnetoDB

Any contribution, comments and pieces of advice are very welcome

Also, I plan to make the lightning talk on Tuesday, November 5 at Expo
Breakout Room 1.

See you on summit.



[1] http://aws.amazon.com/dynamodb

[2] http://aws.amazon.com/dynamodb/testimonials/

[3] http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API.html

[4] http://cassandra.apache.org/

[5] https://blueprints.launchpad.net/trove/+spec/cassandra-db-support

[6] https://review.openstack.org/#/c/51884/

[7]
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/UsingAWSSDK.html

[4] https://github.com/boto/boto

[5] http://boto.readthedocs.org/en/latest/dynamodb2_tut.html

With best regards,
Ilya Sviridov


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev