Re: Apache CloudStack Generic Key-Value and Communication Plug-in

[Ivan Kudryavtsev]

Hello, Daan, Rafael.

Ok, let's discover some points.

Rafael:
All the stuff is managed in Elasticsearch, clustered ES is supported as
well. As for now, value datatype is 'keyword' and doesn't fit well for
blobs. But, actually, there are no practical limitations except we don't
want press the CS server with streaming for blobs as it's design is not
well designed for long lasting jobs like that.

To adress the case we also plan to develop artifact storage which is
intended to keep blobs with versioning and managed security. Probably will
use Ceph under the hood.

Daan:
Let's go with the cases. Actually, in the past I published the announcement
for log management plugin which intention is to provide access to VM logs
from the API and other interfaces. Now we develop CLI utility for it. That
plugin helps to observe what logs are generate during VM deployment.

This plugin (KV) helps to deploy complex topologies, like Docker Swarm,
Kubernetes, MySQL Galera replicaset, DC/OS, monitor the state of the
deployments and do trivial things:

Observe current VM load, RAM availability, disk space. All these parameters
can be observed in historical perspective or last values.

A quorum example. Say, you deploy two VMs programmatically and they have to
decide who is Docker master and who is worker, so you create shared storage
and both VMs place they UUID in 'order' key. They check the history for
'order' and find who is the first. Next, the first generates cluster
initialization and places secret key to 'secret'. After, worker reads it
and joins the cluster. Want to add a new VM to the cluster, just pass that
KV to it as well. Of cource, proper templates with bootstrap code are
required.

Actually, there are tons of very sophisticated deployments can be
implemented in a very generic way with theese two plugins.

Another piece of software to be developed is one-time accessor plugin which
is important for secure sharing of various credentials when multi tenant
VMs are used.

Have a good day.


пт, 21 сент. 2018 г., 2:21 Rafael Weingärtner :

> That is very interesting Ivan!
>
> I have a few questions though:
>
>- Do you have a limit on ”value” size? Can it be a binary of a file?
>- Where do you guys store these “keys” and ”values”?
>
>
> On Wed, Sep 19, 2018 at 4:02 AM, Ivan Kudryavtsev <
> kudryavtsev...@bw-sw.com>
> wrote:
>
> > Hello, community.
> >
> > We glad to announce a new Open Source plug-in for Apache CloudStack which
> > helps to maintain generic key-value storages. There are three types are
> > supported right now:
> >
> > - temporary KV with TTL (many per account);
> > - persistent KV bound to VM (one per VM);
> > - persistent KV bound to account (many per account).
> >
> > KV storages support a commit-log based approach where old values are not
> > erased with new ones and it provides the access to the last key/values
> and
> > to the history which helps to maintain things like quorums and other
> > operations which require serialized access.
> >
> > The plugin uses Elasticsearch. I
> >
> > If you are interested to know more, contact our website for the full
> > announcement and download details:
> > https://bitworks.software/en/2018-09-19-acs-vm-kv-storage-plugin.html
> >
> > All questions, improvements and considerations are welcome!
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks LLC
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ 
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Apache CloudStack Generic Key-Value and Communication Plug-in

[Rafael Weingärtner]

That is very interesting Ivan!

I have a few questions though:

   - Do you have a limit on ”value” size? Can it be a binary of a file?
   - Where do you guys store these “keys” and ”values”?


On Wed, Sep 19, 2018 at 4:02 AM, Ivan Kudryavtsev 
wrote:

> Hello, community.
>
> We glad to announce a new Open Source plug-in for Apache CloudStack which
> helps to maintain generic key-value storages. There are three types are
> supported right now:
>
> - temporary KV with TTL (many per account);
> - persistent KV bound to VM (one per VM);
> - persistent KV bound to account (many per account).
>
> KV storages support a commit-log based approach where old values are not
> erased with new ones and it provides the access to the last key/values and
> to the history which helps to maintain things like quorums and other
> operations which require serialized access.
>
> The plugin uses Elasticsearch. I
>
> If you are interested to know more, contact our website for the full
> announcement and download details:
> https://bitworks.software/en/2018-09-19-acs-vm-kv-storage-plugin.html
>
> All questions, improvements and considerations are welcome!
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ 
>



-- 
Rafael Weingärtner

Re: Apache CloudStack Generic Key-Value and Communication Plug-in

[Daan Hoogland]

sounds nice Ivan, Do you have a whitepaper on it? I strugle to see a clear
usecase for this plugin. maybe you have a casus or two!?!

On Wed, Sep 19, 2018 at 9:02 AM Ivan Kudryavtsev 
wrote:

> Hello, community.
>
> We glad to announce a new Open Source plug-in for Apache CloudStack which
> helps to maintain generic key-value storages. There are three types are
> supported right now:
>
> - temporary KV with TTL (many per account);
> - persistent KV bound to VM (one per VM);
> - persistent KV bound to account (many per account).
>
> KV storages support a commit-log based approach where old values are not
> erased with new ones and it provides the access to the last key/values and
> to the history which helps to maintain things like quorums and other
> operations which require serialized access.
>
> The plugin uses Elasticsearch. I
>
> If you are interested to know more, contact our website for the full
> announcement and download details:
> https://bitworks.software/en/2018-09-19-acs-vm-kv-storage-plugin.html
>
> All questions, improvements and considerations are welcome!
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ 
>


-- 
Daan

Apache CloudStack Generic Key-Value and Communication Plug-in

[Ivan Kudryavtsev]

Hello, community.

We glad to announce a new Open Source plug-in for Apache CloudStack which
helps to maintain generic key-value storages. There are three types are
supported right now:

- temporary KV with TTL (many per account);
- persistent KV bound to VM (one per VM);
- persistent KV bound to account (many per account).

KV storages support a commit-log based approach where old values are not
erased with new ones and it provides the access to the last key/values and
to the history which helps to maintain things like quorums and other
operations which require serialized access.

The plugin uses Elasticsearch. I

If you are interested to know more, contact our website for the full
announcement and download details:
https://bitworks.software/en/2018-09-19-acs-vm-kv-storage-plugin.html

All questions, improvements and considerations are welcome!

-- 
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell: +7-923-414-1515
WWW: http://bitworks.software/