Hi,
I am working on monitoring docker container clusters. It will perform
min/max check per service cluster and scale up/down containers periodically
based on the autoscale policy, instance health and requests in flight.
I figured out the implementations/changes to be done in order to support
docker cluster monitoring.
An initial version of docker cluster monitor (4.1.0 M1) is already merged
into the docker-integration branch.
I will summarize the implementations and changes to be done below.
*Changes to Cluster Monitor Design*
With docker integration, we need to plug in a new docker cluster monitor.
At the moment, we are having the following cluster monitor hierarchy.
We are not able to plug in a new cluster monitor (say container cluster) to
this design. Because, AbstractMonitor is not abstract enough, as it is
based on network partitions. Hence, I propose we should introduce another
level of abstraction so that we will be able to plug in any 'entity'
cluster monitors. Here is the new structure,
WDYT?
Next thing is the auto scaling.
*Changes to Cartridge Agent*
Currently Cartridge Agent is publishing data to CEP via a single stream
(cartridge_agent_health_stats). To support docker scenario, we have two
options,
1. We can introduce an additional parameter to the same stream
(cartridge_agent_health_stats) to identify which type of cluster a
cartridge instance is coming from (VM based or Container based)
Drawbacks - We will be always sending some unwanted properties like
network_partition_id and partition_id
- It will increase overhead in processing events in CEP.
2. We can introduce a separate stream for container (docker) scenario.
WDYT?
*Changes to CEP extension*
Currently CEP is firing events against network partitions. In docker
scenario, it should fire the same events but against clusters. We should
introduce new execution plans for docker scenario, which will process the
data against clusters and fire to the same events.
*Changes to Health Stats Events*
We may need to add an additional attribute, say typeOfCluster, to health
stats events. Then, on receiving events, we would easily identify which
type of cluster the cartridge instance is coming from, and we can retrieve
the attributes we are interested in. Also it makes easier to get the
cluster monitor directly (currently we are searching for cluster monitors
in each cluster monitors' list using cluster id)
*Changes to rules (drools)*
We will be having new rules in drools for docker scenario (it will be
purely based on autoscaling policy, there are no partition/deployment
policies)
Please share your thoughts on this, as these will be implemented in 4.1.0
M2.
Thanks.
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
On Fri, Sep 12, 2014 at 9:49 PM, Akila Ravihansa Perera <raviha...@wso2.com>
wrote:
> Hi,
>
> I've started working on implementing registering Kubernetes endpoint.
> I've shared the Kubernetes information model and sequence diagram for
> registering a Kubernetes endpoint (images attached). Also this can
> viewed on draw.io at [1,2]. We can represent a Kubernetes cluster
> (master and slaves) by KubernetesGroup class which will contain
> KubernetesHost list and KubernetesMaster attributes.
>
> The following REST API endpoints will be exposed to
> register/update/remove Kubernetes clusters. Note that it will
> dynamically register Kubernetes endpoint and persist them in
> auto-scaler registry. Autoscaler will keep an in-memory information
> model of registered Kubernetes clusters and provide web services to
> query and retrieve Kubernetes hosts information.
>
> POST /kubernetes/deploy/group - Creates a KubernetesGroup object
> PUT /kubernetes/deploy/host/{kubernetesGroupId} - Creates a
> KubernetesHost object under the KubernetesGroup object identified by
> kubernetesGroupId
> PUT /kubernetes/update/master - Updates a KubernetesMaster object
> GET /kubernetes/group - Retrieves a list of KubernetesGroup objects
> GET /kubernetes/group/{kubernetesGroupId} - Retrieves the
> KubernetesGroup object identified by kubernetesGroupId
> GET /kubernetes/{kubernetesHostId} - Retrieves the KubernetesHost
> object identified by kubernetesHostId
> DELETE /kubernetes/group/{kubernetesGroupId} - Removes the
> KubernetesGroup object identified by kubernetesGroupId
> DELETE /kubernetes/group/{kubernetesHostId} - Removes the
> KubernetesHost object identified by kubernetesHostId
>
> [1] https://www.draw.io/#G0B4VuDZ50W69_SFozbS1vRVpyOEk
>
> [2] https://www.draw.io/#G0B4VuDZ50W69_d0NPSmFBM2ZsVWs
>
> This is a draft design and might require further improvements. Would
> appreciate your thoughts.
>
> Thanks.
> --
> Akila Ravihansa Perera
> Software Engineer, WSO2
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
