Re: lxd and constraints

2017-01-13 Thread Nate Finch 
I just feel like we're entering a minefield that our application and CLI
aren't really built to handle.  I think we *should* handle it, but it needs
to be well planned out, instead of just doing a tiny piece at a time and
only figuring out later if we did the right thing.

There's a few problems I can see:

1.) you can have 10 lxd containers with memory limit of 2GB on a machine
with 4GB of RAM.  Deploying 10 applications to those containers that each
have a constraint of mem=2GB will not work as you expect.  We could add
extra bookkeeping for this, and warn you that you appear to be
oversubscribing the host, but that's more work.

2.) What happens if you try to deploy a container without a memory limit on
a host that already has a container on it?

For example:
4GB host
2GB lxd container
try to deploy a new service in a container on this machine.
Do we warn?  We have no clue how much RAM the service will use.  Maybe
it'll be fine, maybe it won't.

3.) Our CLI doesn't really work well with constraints on containers:

juju deploy mysql --constraints mem=2G --to lxd

Does this deploy a machine with 2GB of ram and a container with a 2GB ram
limit on it?  Or does it deploy a machine with 2GB of ram and a container
with no limit on it?  It has to be one or the other, and currently we have
no way of indicating which we want to do, and no way to do the other one
without using multiple commands.

This is a more likely use case, creating a bigger machine that can hold
multiple containers:
juju add-machine --constraints mem=4GB
// adds machine, let's say 5
// create a container on machine 5 with 2GB memory limit
juju deploy mysql --constraints mem=2GB --to lxd:5

At least in this case, the deploy command is clear, there's only one thing
they can possibly mean.  Usually, the placement directive would override
the constraint, but in this case, it does what you would want... but it is
a littler weird that --to lxd:5 uses the constraint, but --to 5 ignores it.

Note that you can't just write a simple script to do the above, because the
machine number is variable, so you have to parse our output and then use
that for the next command.  It's still scriptable, obviously, but it's more
complicated script than just two lines of bash.

Also note that using this second method, you can't deploy more than one
unit at a time, unless you want multiple units on containers on the same
machine (which I think would be pretty odd).



On Fri, Jan 13, 2017 at 3:48 AM Rick Harding 
wrote:

In the end, you say you want an instance with 2gb of ram and if the cloud
has an instance with that exact limit it is in fact an exact limit. The key
things here is the clouds don't have infinite malleable instance types like
containers (this works for kvm and for lxd). So I'm not sure the mis-match
is as far apart as it seems. root disk means give me a disk this big, if
you ask for 2 core as long as you can match an instance type with 2 cores
it's exactly the max you get.

It seems part of this can be more adjusting the language from "minimum" to
something closer to "requested X" where request gives it more of a "I want
X" without the min/max boundaries.



On Fri, Jan 13, 2017 at 3:14 AM John Meinel  wrote:

So we could make it so that constraints are actually 'exactly' for LXD,
which would then conform to both minimum and maximum, and would still be
actually useful for people deploying to containers. We could certainly
probe the host machine and say "you asked for 48 cores, and the host
machine doesn't have it".

However, note that explicit placement also takes precedence over
constraints anyway. If you do:
  juju deploy mysql --constraints mem=4G
today, and then do:
 juju add-unit --to 2
We don't apply the constraint limitations to that specific unit. Arguably
we should at *least* be warning that the constraints for the overall
application don't appear to be valid for this instance.

I guess I'd rather see constraints still set limits for containers, because
people really want that functionality, and that we warn any time you do a
direct placement and the constraints aren't satisfied. (but warn isn't
failing the attempt)

John
=:->

On Fri, Jan 13, 2017 at 10:09 AM, Stuart Bishop  wrote:

On 13 January 2017 at 02:20, Nate Finch  wrote:

I'm implementing constraints for lxd containers and provider... and
stumbled on an impedance mismatch that I don't know how to handle.



I'm not really sure how to resolve this problem.  Maybe it's not a
problem.  Maybe constraints just have a different meaning for containers?
You have to specify the machine number you're deploying to for any
deployment past the first anyway, so you're already manually choosing the
machine, at which point, constraints don't really make sense anyway.


I don't think Juju can handle this. Either constraints have different
meanings with different cloud providers, or lxd needs

Re: Squelching Leadership messages

2017-01-13 Thread Katherine Cox-Buday 
John A Meinel  writes:

> Note that we should still report leadership changes at INFO level, which 
> should allow you to
> debug easy things without changing the log level, as those changes should 
> happen very
> rarely.

+1, thanks for this!

-- 
Katherine

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Squelching Leadership messages

2017-01-13 Thread John A Meinel 
I just proposed 2 branches, one for 2.1 and one for develop. So that the
"will renew leadership" message that triggers every 30s is dropped to DEBUG
level instead of INFO level. That should make those logs have a much higher
signal to noise ratio. But I want to make sure that if anyone is trying to
debug Leadership issues in their charms, they will likely want to use DEBUG
level.

Note that we should still report leadership changes at INFO level, which
should allow you to debug easy things without changing the log level, as
those changes should happen very rarely.

John
=:->

PRs>
 https://github.com/juju/juju/pull/6797
 https://github.com/juju/juju/pull/6798


This is what 'juju debug-log --include-module juju.worker.leadership' looks
like now:
unit-ul-3: 16:51:51 DEBUG juju.worker.leadership ul/3 making initial claim
for ul leadership
unit-ul-3: 16:51:51 INFO juju.worker.leadership ul leadership for ul/3
denied
unit-ul-3: 16:51:51 DEBUG juju.worker.leadership ul/3 waiting for ul
leadership release
unit-ul-3: 16:51:51 DEBUG juju.worker.leadership ul/3 is not ul leader
unit-ul-4: 16:52:01 DEBUG juju.worker.leadership ul/4 making initial claim
for ul leadership
unit-ul-4: 16:52:01 INFO juju.worker.leadership ul leadership for ul/4
denied
unit-ul-4: 16:52:01 DEBUG juju.worker.leadership ul/4 waiting for ul
leadership release
unit-ul-4: 16:52:01 DEBUG juju.worker.leadership ul/4 is not ul leader
unit-ul-4: 16:53:27 INFO juju.worker.leadership ul/4 promoted to leadership
of ul
unit-ul-3: 16:53:27 INFO juju.worker.leadership ul leadership for ul/3
denied
unit-ul-3: 16:53:27 DEBUG juju.worker.leadership ul/3 waiting for ul
leadership release
unit-ul-2: 16:53:40 DEBUG juju.worker.leadership ul/2 making initial claim
for ul leadership
unit-ul-2: 16:53:40 INFO juju.worker.leadership ul leadership for ul/2
denied
unit-ul-2: 16:53:40 DEBUG juju.worker.leadership ul/2 waiting for ul
leadership release
unit-ul-2: 16:53:41 DEBUG juju.worker.leadership ul/2 is not ul leader
unit-ul-3: 16:54:26 DEBUG juju.worker.leadership ul/3 making initial claim
for ul leadership
unit-ul-3: 16:54:26 INFO juju.worker.leadership ul leadership for ul/3
denied
unit-ul-3: 16:54:26 DEBUG juju.worker.leadership ul/3 waiting for ul
leadership release
unit-ul-3: 16:54:26 DEBUG juju.worker.leadership ul/3 is not ul leader
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: lxd and constraints

2017-01-13 Thread Rick Harding 
In the end, you say you want an instance with 2gb of ram and if the cloud
has an instance with that exact limit it is in fact an exact limit. The key
things here is the clouds don't have infinite malleable instance types like
containers (this works for kvm and for lxd). So I'm not sure the mis-match
is as far apart as it seems. root disk means give me a disk this big, if
you ask for 2 core as long as you can match an instance type with 2 cores
it's exactly the max you get.

It seems part of this can be more adjusting the language from "minimum" to
something closer to "requested X" where request gives it more of a "I want
X" without the min/max boundaries.



On Fri, Jan 13, 2017 at 3:14 AM John Meinel  wrote:

> So we could make it so that constraints are actually 'exactly' for LXD,
> which would then conform to both minimum and maximum, and would still be
> actually useful for people deploying to containers. We could certainly
> probe the host machine and say "you asked for 48 cores, and the host
> machine doesn't have it".
>
> However, note that explicit placement also takes precedence over
> constraints anyway. If you do:
>   juju deploy mysql --constraints mem=4G
> today, and then do:
>  juju add-unit --to 2
> We don't apply the constraint limitations to that specific unit. Arguably
> we should at *least* be warning that the constraints for the overall
> application don't appear to be valid for this instance.
>
> I guess I'd rather see constraints still set limits for containers,
> because people really want that functionality, and that we warn any time
> you do a direct placement and the constraints aren't satisfied. (but warn
> isn't failing the attempt)
>
> John
> =:->
>
> On Fri, Jan 13, 2017 at 10:09 AM, Stuart Bishop <
> stuart.bis...@canonical.com> wrote:
>
> On 13 January 2017 at 02:20, Nate Finch  wrote:
>
> I'm implementing constraints for lxd containers and provider... and
> stumbled on an impedance mismatch that I don't know how to handle.
>
>
>
> I'm not really sure how to resolve this problem.  Maybe it's not a
> problem.  Maybe constraints just have a different meaning for containers?
> You have to specify the machine number you're deploying to for any
> deployment past the first anyway, so you're already manually choosing the
> machine, at which point, constraints don't really make sense anyway.
>
>
> I don't think Juju can handle this. Either constraints have different
> meanings with different cloud providers, or lxd needs to accept minimum
> constraints (along with any other cloud providers with this behavior).
>
> If you decide constraints need to consistently mean minimum, then I'd
> argue it is best to not pass them to current-gen lxd at all. Enforcing that
> containers are restricted to the minimum viable resources declared in a
> bundle does not seem helpful, and Juju does not have enough information to
> choose suitable maximums (and if it did, would not know if they would
> remain suitable tomorrow).
>
> --
> Stuart Bishop 
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: lxd and constraints

2017-01-13 Thread John Meinel 
So we could make it so that constraints are actually 'exactly' for LXD,
which would then conform to both minimum and maximum, and would still be
actually useful for people deploying to containers. We could certainly
probe the host machine and say "you asked for 48 cores, and the host
machine doesn't have it".

However, note that explicit placement also takes precedence over
constraints anyway. If you do:
  juju deploy mysql --constraints mem=4G
today, and then do:
 juju add-unit --to 2
We don't apply the constraint limitations to that specific unit. Arguably
we should at *least* be warning that the constraints for the overall
application don't appear to be valid for this instance.

I guess I'd rather see constraints still set limits for containers, because
people really want that functionality, and that we warn any time you do a
direct placement and the constraints aren't satisfied. (but warn isn't
failing the attempt)

John
=:->

On Fri, Jan 13, 2017 at 10:09 AM, Stuart Bishop  wrote:

> On 13 January 2017 at 02:20, Nate Finch  wrote:
>
> I'm implementing constraints for lxd containers and provider... and
>> stumbled on an impedance mismatch that I don't know how to handle.
>>
>
>
>> I'm not really sure how to resolve this problem.  Maybe it's not a
>> problem.  Maybe constraints just have a different meaning for containers?
>> You have to specify the machine number you're deploying to for any
>> deployment past the first anyway, so you're already manually choosing the
>> machine, at which point, constraints don't really make sense anyway.
>>
>
> I don't think Juju can handle this. Either constraints have different
> meanings with different cloud providers, or lxd needs to accept minimum
> constraints (along with any other cloud providers with this behavior).
>
> If you decide constraints need to consistently mean minimum, then I'd
> argue it is best to not pass them to current-gen lxd at all. Enforcing that
> containers are restricted to the minimum viable resources declared in a
> bundle does not seem helpful, and Juju does not have enough information to
> choose suitable maximums (and if it did, would not know if they would
> remain suitable tomorrow).
>
> --
> Stuart Bishop 
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: lxd and constraints

2017-01-13 Thread Stuart Bishop 
On 13 January 2017 at 02:20, Nate Finch  wrote:

I'm implementing constraints for lxd containers and provider... and
> stumbled on an impedance mismatch that I don't know how to handle.
>


> I'm not really sure how to resolve this problem.  Maybe it's not a
> problem.  Maybe constraints just have a different meaning for containers?
> You have to specify the machine number you're deploying to for any
> deployment past the first anyway, so you're already manually choosing the
> machine, at which point, constraints don't really make sense anyway.
>

I don't think Juju can handle this. Either constraints have different
meanings with different cloud providers, or lxd needs to accept minimum
constraints (along with any other cloud providers with this behavior).

If you decide constraints need to consistently mean minimum, then I'd argue
it is best to not pass them to current-gen lxd at all. Enforcing that
containers are restricted to the minimum viable resources declared in a
bundle does not seem helpful, and Juju does not have enough information to
choose suitable maximums (and if it did, would not know if they would
remain suitable tomorrow).

-- 
Stuart Bishop 
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev