Re: Juju and cluster managers (like mesos)

[Merlijn Sebrechts]

This is exactly what I was looking for! Thank you so much for this! This is
a great way to integrate Juju with Mesos so you get the best of both
worlds...

2015-12-01 12:45 GMT+01:00 Nicolas Thomas  :

>
>
> On Mon, Nov 30, 2015 at 7:54 PM, Merlijn Sebrechts <
> merlijn.sebrec...@gmail.com> wrote:
>
>> Hi Samuel
>>
>>
>> Thanks for your answer!
>>
>> The injection Charm sounds interesting. So Juju manages both k8s and the
>> services running on top of k8s? Do you have an example you can show me?
>>
>
> ​Better than that  a full "dangerous demo"
> https://www.youtube.com/watch?v=Q_370CUw5Yk
>
> And the corresponding bundle to test it on any cloud:
> https://jujucharms.com/u/tads2015dataart/tads2015-demo/21
>
> ​The only thing (provided by Truephone) missing is the actual phone number
> to the platform link (you can use SIP or SIP Trunk service supported by
> Telscale)..
>
> Enjoy​
>
>
>
>> > However, monitoring is an expression of the operation and not the
>> model, therefore "can not" be operated by Juju.
>>
>> Can you explain this a bit more? What do you see as model, what do you
>> see as operation, and how do actions and hooks fit into this?
>>
>>
> ​The true real life monitoring is best done from the workload perspective
> (sessions, trends, link to generic cpu/ram metrics) .. It comes after you
> model your architecture and must not dictate the way you model.
>
> BUT Juju have an API allowing to use cleanly modelled scale out
> capabilities in Juju to react to your monitoring (this can vary from test,
> benchmark, prod, etc...) this is what the demo does if Mesos start to reach
> the limits juju extend mesos (and dependencies), same if way underused..  ​
>
> ​Hope this makes the intent clearer..​
>
>
>
>> > Scaling should not depend on Juju either in our current vision. It's
>> not an expression of a model, but rather of how to operate the model.
>>
>> If I understand correctly, the model (Charms) should specify how a
>> service can scale, and an external entity (the user or another service)
>> should specify when those scaling actions should be called?
>>
>> I'd like for bundles to be able to specify a lot more about the
>> "macroservice" they deploy. So you could run "juju add-unit" on a bundle,
>> and it would scale the required services. Ofcourse, this will have to be a
>> lot "smarter" because each macroservice has a number of different ways to
>> scale. I think a lot of this functionality can be accomplished if a bundle
>> would be able to have actions or hooks... Another use-case would be to
>> upgrade all charms in a bundle to versions that are known to work together,
>> although I think this is currently possible using the deployer...
>>
>>
>>
>> Kind regards
>> Merlijn Sebrechts
>>
>>
>> 2015-11-30 16:13 GMT+01:00 Samuel Cozannet > >:
>>
>>> Adding Artyom from DataArt who has a lot of very clever ideas about this
>>> and recently created a cool autoscaling demo.
>>>
>>> I'm interested in any follow up given to your work. I share your
>>> frustration at containers systems using a so called "orchestration" when
>>> the orchestration is really some basic hooks.
>>> The consequence of these systems is a total absence of portability
>>> between techs (moving from k8s to Swarm or worse Mesos requires a lot of
>>> rewriting the core and even sometimes rebuild some of the containers to
>>> adapt to the service discovery APIs). Something that Juju wants to address
>>> really well.
>>>
>>> My path so far is to create specific injection charms for k8s and others
>>> (Swarm so far). By talking only to the current leader, you kind of create
>>> this abstraction you are talking about.
>>> That means you can then expose configuration to scale out & in the
>>> service by calling the Juju API to reconfigure the service itself.
>>> Not a complete solution, but a starting point. The issue with it is that
>>> to comply with Juju models, I have to create an injection charm per app,
>>> which is additional work on top of containerizing for example.
>>> The LXD provider will certainly help in that space, even more when/if
>>> LXD become first class citizen in "Container Orchestration Tools".
>>>
>>> As for monitoring, some charms expose monitoring hooks that can be
>>> consumed by other specialized services. As a consequence you can easily
>>> integrate not only with service spawned by Juju, but also external systems.
>>> However, monitoring is an expression of the operation and not the model,
>>> therefore "can not" be operated by Juju.
>>>
>>> Scaling should not depend on Juju either in our current vision. It's not
>>> an expression of a model, but rather of how to operate the model.
>>> Therefore, this task should stay outside of Juju, even if it can be
>>> operated via Juju's APIs (scale out / in, potentially rolling upgrades in
>>> the future).
>>>
>>> ++
>>> Sam
>>>
>>>
>>>
>>>
>>> --
>>> Samuel Cozannet
>>> Cloud, Big Data and IoT Strategy Team
>>> Business Development - Cloud and ISV Ecosystem
>>> Changin

Re: Juju and cluster managers (like mesos)

[Nicolas Thomas ]

On Mon, Nov 30, 2015 at 7:54 PM, Merlijn Sebrechts <
merlijn.sebrec...@gmail.com> wrote:

> Hi Samuel
>
>
> Thanks for your answer!
>
> The injection Charm sounds interesting. So Juju manages both k8s and the
> services running on top of k8s? Do you have an example you can show me?
>

​Better than that  a full "dangerous demo"
https://www.youtube.com/watch?v=Q_370CUw5Yk

And the corresponding bundle to test it on any cloud:
https://jujucharms.com/u/tads2015dataart/tads2015-demo/21

​The only thing (provided by Truephone) missing is the actual phone number
to the platform link (you can use SIP or SIP Trunk service supported by
Telscale)..

Enjoy​



> > However, monitoring is an expression of the operation and not the
> model, therefore "can not" be operated by Juju.
>
> Can you explain this a bit more? What do you see as model, what do you see
> as operation, and how do actions and hooks fit into this?
>
>
​The true real life monitoring is best done from the workload perspective
(sessions, trends, link to generic cpu/ram metrics) .. It comes after you
model your architecture and must not dictate the way you model.

BUT Juju have an API allowing to use cleanly modelled scale out
capabilities in Juju to react to your monitoring (this can vary from test,
benchmark, prod, etc...) this is what the demo does if Mesos start to reach
the limits juju extend mesos (and dependencies), same if way underused..  ​

​Hope this makes the intent clearer..​



> > Scaling should not depend on Juju either in our current vision. It's
> not an expression of a model, but rather of how to operate the model.
>
> If I understand correctly, the model (Charms) should specify how a service
> can scale, and an external entity (the user or another service) should
> specify when those scaling actions should be called?
>
> I'd like for bundles to be able to specify a lot more about the
> "macroservice" they deploy. So you could run "juju add-unit" on a bundle,
> and it would scale the required services. Ofcourse, this will have to be a
> lot "smarter" because each macroservice has a number of different ways to
> scale. I think a lot of this functionality can be accomplished if a bundle
> would be able to have actions or hooks... Another use-case would be to
> upgrade all charms in a bundle to versions that are known to work together,
> although I think this is currently possible using the deployer...
>
>
>
> Kind regards
> Merlijn Sebrechts
>
>
> 2015-11-30 16:13 GMT+01:00 Samuel Cozannet 
> :
>
>> Adding Artyom from DataArt who has a lot of very clever ideas about this
>> and recently created a cool autoscaling demo.
>>
>> I'm interested in any follow up given to your work. I share your
>> frustration at containers systems using a so called "orchestration" when
>> the orchestration is really some basic hooks.
>> The consequence of these systems is a total absence of portability
>> between techs (moving from k8s to Swarm or worse Mesos requires a lot of
>> rewriting the core and even sometimes rebuild some of the containers to
>> adapt to the service discovery APIs). Something that Juju wants to address
>> really well.
>>
>> My path so far is to create specific injection charms for k8s and others
>> (Swarm so far). By talking only to the current leader, you kind of create
>> this abstraction you are talking about.
>> That means you can then expose configuration to scale out & in the
>> service by calling the Juju API to reconfigure the service itself.
>> Not a complete solution, but a starting point. The issue with it is that
>> to comply with Juju models, I have to create an injection charm per app,
>> which is additional work on top of containerizing for example.
>> The LXD provider will certainly help in that space, even more when/if LXD
>> become first class citizen in "Container Orchestration Tools".
>>
>> As for monitoring, some charms expose monitoring hooks that can be
>> consumed by other specialized services. As a consequence you can easily
>> integrate not only with service spawned by Juju, but also external systems.
>> However, monitoring is an expression of the operation and not the model,
>> therefore "can not" be operated by Juju.
>>
>> Scaling should not depend on Juju either in our current vision. It's not
>> an expression of a model, but rather of how to operate the model.
>> Therefore, this task should stay outside of Juju, even if it can be
>> operated via Juju's APIs (scale out / in, potentially rolling upgrades in
>> the future).
>>
>> ++
>> Sam
>>
>>
>>
>>
>> --
>> Samuel Cozannet
>> Cloud, Big Data and IoT Strategy Team
>> Business Development - Cloud and ISV Ecosystem
>> Changing the Future of Cloud
>> Ubuntu   / Canonical UK LTD  /
>> Juju 
>> samuel.cozan...@canonical.com
>> mob: +33 616 702 389
>> skype: samnco
>> Twitter: @SaMnCo_23
>> [image: View Samuel Cozannet's profile on LinkedIn]
>> 
>>
>> On Mon, Nov

Re: Juju and cluster managers (like mesos)

[Samuel Cozannet]

Hey,

This is very much early stage and a demo I am still working on to build an
IoT pipeline. Most of it unpublished yet and unstable. Will share as soon
as I have something more consistent (if you are interested even now let me
know and we'll sort out something)
This work actually uses Swarm (as it was primarily intended as a DockerCon
talk), but yes, the idea is that Juju:
* Deploys a Swarm cluster (in the future s/Swarm/{k8s, Mesos, ...})
* Deploys (currently subordinate, but colocation would actually be a better
way, I need to change that) a docker-compose charm that will deploy on the
cluster (in our first case a rabbitmq server)
* Scans the docker-compose file for ports to open for the workloads, and
open-port them
* Presents the interfaces needed (copy / paste from the charm that installs
rabbitmq)
==> running in a container in a cluster or in a VM/bare metal as a unit is
"the same" from a Juju perspective.

For now,
* In this specific case, I only present the amqp interface, the other
relations are not exposed.
* In a perfect world, it also  means the maintainer has to keep in sync
with upstream (if one exists), or that the charms is clever enough to know
if it's supposed to deploy in a containerized env or not.
* The number of units of the charm = number of units of the target cluster.
Which means every connecting charm will have to manage leadership
* Many other problems I have not thought through yet, gladly welcoming
thoughts :)

Model is essentially any aspect of the architecture that is not modified by
operations. It's the list of services and their relations, regardless of
their individual size. For example, Dev, Staging and Prod share the same
model.

Operations encompasses anything that is set as a reaction to the
environment, or a human action to modify its behaviour at runtime. Scaling,
scaling rules & triggers, monitoring tools (...) are elements that differ
depending on how / where the model is deployed.

Scaling is in many cases a rule of thumb arbitrarily set by the ops team
and based on their experience and expectations.
For example, one could want to scale out when CPU load exceeds 70% in
average on all units, while another could set that trigger at 80%.

As a consequence, it is not something that should be taken care of by Juju,
as Juju only presents the model, that can be deployed in a fast,
repeatable, reliable and observable way.

You'll notice that similarly you don't set autoscaling triggers in k8s,
mesos or Swarm. You still have to call the API. I guess it is because of
the same conception that the admin's knowledge and know-how should drive
that, not the tool.

Ben Saller is the the most knowledgeable person at Canonical about current
work on your last bits. You may want to exchange a couple of off thread
emails with him about it :)

++
Sam







--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu   / Canonical UK LTD  / Juju

samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]


On Mon, Nov 30, 2015 at 7:54 PM, Merlijn Sebrechts <
merlijn.sebrec...@gmail.com> wrote:

> Hi Samuel
>
>
> Thanks for your answer!
>
> The injection Charm sounds interesting. So Juju manages both k8s and the
> services running on top of k8s? Do you have an example you can show me?
>
> > However, monitoring is an expression of the operation and not the
> model, therefore "can not" be operated by Juju.
>
> Can you explain this a bit more? What do you see as model, what do you see
> as operation, and how do actions and hooks fit into this?
>
> > Scaling should not depend on Juju either in our current vision. It's
> not an expression of a model, but rather of how to operate the model.
>
> If I understand correctly, the model (Charms) should specify how a service
> can scale, and an external entity (the user or another service) should
> specify when those scaling actions should be called?
>
> I'd like for bundles to be able to specify a lot more about the
> "macroservice" they deploy. So you could run "juju add-unit" on a bundle,
> and it would scale the required services. Ofcourse, this will have to be a
> lot "smarter" because each macroservice has a number of different ways to
> scale. I think a lot of this functionality can be accomplished if a bundle
> would be able to have actions or hooks... Another use-case would be to
> upgrade all charms in a bundle to versions that are known to work together,
> although I think this is currently possible using the deployer...
>
>
>
> Kind regards
> Merlijn Sebrechts
>
>
> 2015-11-30 16:13 GMT+01:00 Samuel Cozannet 
> :
>
>> Adding Artyom from DataArt who has a lot of very clever ideas about this
>> and recently created a cool autoscaling demo.
>>
>> I'm interested in any follow up give

Re: Juju and cluster managers (like mesos)

[Merlijn Sebrechts]

Hi Samuel


Thanks for your answer!

The injection Charm sounds interesting. So Juju manages both k8s and the
services running on top of k8s? Do you have an example you can show me?

> However, monitoring is an expression of the operation and not the model,
therefore "can not" be operated by Juju.

Can you explain this a bit more? What do you see as model, what do you see
as operation, and how do actions and hooks fit into this?

> Scaling should not depend on Juju either in our current vision. It's not
an expression of a model, but rather of how to operate the model.

If I understand correctly, the model (Charms) should specify how a service
can scale, and an external entity (the user or another service) should
specify when those scaling actions should be called?

I'd like for bundles to be able to specify a lot more about the
"macroservice" they deploy. So you could run "juju add-unit" on a bundle,
and it would scale the required services. Ofcourse, this will have to be a
lot "smarter" because each macroservice has a number of different ways to
scale. I think a lot of this functionality can be accomplished if a bundle
would be able to have actions or hooks... Another use-case would be to
upgrade all charms in a bundle to versions that are known to work together,
although I think this is currently possible using the deployer...



Kind regards
Merlijn Sebrechts


2015-11-30 16:13 GMT+01:00 Samuel Cozannet :

> Adding Artyom from DataArt who has a lot of very clever ideas about this
> and recently created a cool autoscaling demo.
>
> I'm interested in any follow up given to your work. I share your
> frustration at containers systems using a so called "orchestration" when
> the orchestration is really some basic hooks.
> The consequence of these systems is a total absence of portability between
> techs (moving from k8s to Swarm or worse Mesos requires a lot of rewriting
> the core and even sometimes rebuild some of the containers to adapt to the
> service discovery APIs). Something that Juju wants to address really well.
>
> My path so far is to create specific injection charms for k8s and others
> (Swarm so far). By talking only to the current leader, you kind of create
> this abstraction you are talking about.
> That means you can then expose configuration to scale out & in the service
> by calling the Juju API to reconfigure the service itself.
> Not a complete solution, but a starting point. The issue with it is that
> to comply with Juju models, I have to create an injection charm per app,
> which is additional work on top of containerizing for example.
> The LXD provider will certainly help in that space, even more when/if LXD
> become first class citizen in "Container Orchestration Tools".
>
> As for monitoring, some charms expose monitoring hooks that can be
> consumed by other specialized services. As a consequence you can easily
> integrate not only with service spawned by Juju, but also external systems.
> However, monitoring is an expression of the operation and not the model,
> therefore "can not" be operated by Juju.
>
> Scaling should not depend on Juju either in our current vision. It's not
> an expression of a model, but rather of how to operate the model.
> Therefore, this task should stay outside of Juju, even if it can be
> operated via Juju's APIs (scale out / in, potentially rolling upgrades in
> the future).
>
> ++
> Sam
>
>
>
>
> --
> Samuel Cozannet
> Cloud, Big Data and IoT Strategy Team
> Business Development - Cloud and ISV Ecosystem
> Changing the Future of Cloud
> Ubuntu   / Canonical UK LTD  /
> Juju 
> samuel.cozan...@canonical.com
> mob: +33 616 702 389
> skype: samnco
> Twitter: @SaMnCo_23
> [image: View Samuel Cozannet's profile on LinkedIn]
> 
>
> On Mon, Nov 30, 2015 at 2:47 PM, Merlijn Sebrechts <
> merlijn.sebrec...@gmail.com> wrote:
>
>> Hi all
>>
>>
>> I'd like to start a discussion about how Juju can fit into cluster
>> managers like Apache Mesos and Kubernetes.
>>
>> Currently, Juju fits nicely into this story as a way to setup these
>> cluster managers. Payloads continue on that idea with Juju as a manager of
>> a cluster manager. However, I'm a lot more interested in Juju on top of a
>> cluster manager, where the cluster manager would be a provider Juju uses to
>> deploy services on.
>>
>> Juju provides an awesome way to model complex services in a modular and
>> re-usable way. The relationships allow for much more complex interactions
>> between services than what the "service discovery" in Kubernetes and Mesos
>> allows. Service discovery allows for a service to say "I need the IP's of
>> these services" but that's pretty much it. No flexible adaptable
>> infrastructure where services change their behavior depending on what they
>> are connected to. It basically stems from the same mindset that brought us
>> tools like Chef and Puppet: One company with a big d

Re: Juju and cluster managers (like mesos)

[Samuel Cozannet]

Adding Artyom from DataArt who has a lot of very clever ideas about this
and recently created a cool autoscaling demo.

I'm interested in any follow up given to your work. I share your
frustration at containers systems using a so called "orchestration" when
the orchestration is really some basic hooks.
The consequence of these systems is a total absence of portability between
techs (moving from k8s to Swarm or worse Mesos requires a lot of rewriting
the core and even sometimes rebuild some of the containers to adapt to the
service discovery APIs). Something that Juju wants to address really well.

My path so far is to create specific injection charms for k8s and others
(Swarm so far). By talking only to the current leader, you kind of create
this abstraction you are talking about.
That means you can then expose configuration to scale out & in the service
by calling the Juju API to reconfigure the service itself.
Not a complete solution, but a starting point. The issue with it is that to
comply with Juju models, I have to create an injection charm per app, which
is additional work on top of containerizing for example.
The LXD provider will certainly help in that space, even more when/if LXD
become first class citizen in "Container Orchestration Tools".

As for monitoring, some charms expose monitoring hooks that can be consumed
by other specialized services. As a consequence you can easily integrate
not only with service spawned by Juju, but also external systems. However,
monitoring is an expression of the operation and not the model, therefore
"can not" be operated by Juju.

Scaling should not depend on Juju either in our current vision. It's not an
expression of a model, but rather of how to operate the model. Therefore,
this task should stay outside of Juju, even if it can be operated via
Juju's APIs (scale out / in, potentially rolling upgrades in the future).

++
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu   / Canonical UK LTD  / Juju

samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]


On Mon, Nov 30, 2015 at 2:47 PM, Merlijn Sebrechts <
merlijn.sebrec...@gmail.com> wrote:

> Hi all
>
>
> I'd like to start a discussion about how Juju can fit into cluster
> managers like Apache Mesos and Kubernetes.
>
> Currently, Juju fits nicely into this story as a way to setup these
> cluster managers. Payloads continue on that idea with Juju as a manager of
> a cluster manager. However, I'm a lot more interested in Juju on top of a
> cluster manager, where the cluster manager would be a provider Juju uses to
> deploy services on.
>
> Juju provides an awesome way to model complex services in a modular and
> re-usable way. The relationships allow for much more complex interactions
> between services than what the "service discovery" in Kubernetes and Mesos
> allows. Service discovery allows for a service to say "I need the IP's of
> these services" but that's pretty much it. No flexible adaptable
> infrastructure where services change their behavior depending on what they
> are connected to. It basically stems from the same mindset that brought us
> tools like Chef and Puppet: One company with a big dev team that creates
> services for internal use only.
>
>
>- Cluster managers are very good at scheduling "dumb" workloads.
>They're a datacenter kernel, they don't care what runs in the container. At
>best, they provide a way for two containers to communicate (service
>discovery).
>
>
>- Juju is very good at configuring applications. It changes services
>depending on how they are connected. Juju for the most part doesn't care
>where services run, only how they are connected.
>
>
> Combine these two and you could get an awesome PaaS that can run a lot
> more than "dumb" 12-factor apps. It bothers me to see frameworks like
> Kubernetes use the terminology "service orchestration" when all they do is
> connect static services. Juju is on to something with its dynamic
> relations, but it seems not many people have caught on...
>
> I see two possible paths to integrate Service Orchestration with Cluster
> management (scheduling):
>
>
>- *Support cluster managers in Juju as providers.* This might be hard
>to do since Juju's units require an OS-level container, not a process
>container. Step 1 would be LXD support in Mesos/Kubernetes?
>
>
>- *Write cluster management extensions on top of Juju.* Basically
>recreate the scheduling, failover and scaling functionality of
>Kubernetes/Mesos in Juju. There seem to be some people in the Juju
>ecosystem who are working on their own version of this. I've seen some
>people who are trying to automate the up/down scaling of services. Maybe 

Juju and cluster managers (like mesos)

[Merlijn Sebrechts]

Hi all


I'd like to start a discussion about how Juju can fit into cluster managers
like Apache Mesos and Kubernetes.

Currently, Juju fits nicely into this story as a way to setup these cluster
managers. Payloads continue on that idea with Juju as a manager of a
cluster manager. However, I'm a lot more interested in Juju on top of a
cluster manager, where the cluster manager would be a provider Juju uses to
deploy services on.

Juju provides an awesome way to model complex services in a modular and
re-usable way. The relationships allow for much more complex interactions
between services than what the "service discovery" in Kubernetes and Mesos
allows. Service discovery allows for a service to say "I need the IP's of
these services" but that's pretty much it. No flexible adaptable
infrastructure where services change their behavior depending on what they
are connected to. It basically stems from the same mindset that brought us
tools like Chef and Puppet: One company with a big dev team that creates
services for internal use only.


   - Cluster managers are very good at scheduling "dumb" workloads. They're
   a datacenter kernel, they don't care what runs in the container. At best,
   they provide a way for two containers to communicate (service discovery).


   - Juju is very good at configuring applications. It changes services
   depending on how they are connected. Juju for the most part doesn't care
   where services run, only how they are connected.


Combine these two and you could get an awesome PaaS that can run a lot more
than "dumb" 12-factor apps. It bothers me to see frameworks like Kubernetes
use the terminology "service orchestration" when all they do is connect
static services. Juju is on to something with its dynamic relations, but it
seems not many people have caught on...

I see two possible paths to integrate Service Orchestration with Cluster
management (scheduling):


   - *Support cluster managers in Juju as providers.* This might be hard to
   do since Juju's units require an OS-level container, not a process
   container. Step 1 would be LXD support in Mesos/Kubernetes?


   - *Write cluster management extensions on top of Juju.* Basically
   recreate the scheduling, failover and scaling functionality of
   Kubernetes/Mesos in Juju. There seem to be some people in the Juju
   ecosystem who are working on their own version of this. I've seen some
   people who are trying to automate the up/down scaling of services. Maybe it
   might be good to bundle these efforts and make a generic framework for
   this?

Juju can encapsulate installation and configuration best-practices.
Wouldn't it be great if Juju could encapsulate monitoring, scaling and
scheduling best-practices too? Maybe by making bundles smarter? (Just
imagine: a Hadoop bundle with builtin monitoring and scaling that scales
the workers up/down depending on the need)

*Now I'm curious to see what you guys think about this. *Do you think there
is a need to integrate Juju with "datacenter kernels"? What is Canonical's
vision about this? What does the community think?



Kind regards
Merlijn Sebrechts
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju