+1 (million), this solution keeps coming up, and I still feel it is
the right one.
On Thu, Nov 7, 2013 at 7:07 AM, Kapil Thangavelu
<kapil.thangav...@canonical.com> wrote:
>
>
>
> On Thu, Nov 7, 2013 at 2:49 AM, roger peppe <rogpe...@gmail.com> wrote:
>>
>> The current plan is to have a single "juju ensure-ha-state" juju
>> command. This would create new state server machines if there are less
>> than the required number (currently 3).
>>
>> Taking that as given, I'm wondering what we should do
>> in the future, when users require more than a single
>> big On switch for HA.
>>
>> How does the user:
>>
>> a) know about the HA machines so the costs of HA are not hidden, and that
>> the implications of particular machine failures are clear?
>>
>> b) fix the system when a machine dies?
>>
>> c) scale up the system to x thousand nodes?
>>
>> d) scale down the system?
>>
>> For a), we could tag a machine in the status as a "state server", and
>> hope that the user knows what that means.
>>
>> For b) the suggestion is that the user notice that a state server machine
>> is non-responsive (as marked in status) and runs destroy-machine on it,
>> which will notice that it's a state server machine and automatically
>> start another one to replace it. Destroy-machine would refuse to work
>> on a state server machine that seems to be alive.
>>
>> For c) we could add a flag to ensure-ha-state suggesting a desired number
>> of state-server nodes.
>>
>> I'm not sure what the suggestion is for d) given that we refuse to
>> destroy live state-server machines.
>>
>> Although ensure-ha-state might be a fine way to turn
>> on HA initially I'm not entirely happy with expanding it to cover
>> all the above cases. It seems to me like we're going
>> to create a leaky abstraction that purports to be magic ("just wave the
>> HA wand!") and ends up being limiting, and in some cases confusing
>> ("Huh? I asked to destroy that machine and there's another one
>> just been created")
>>
>> I believe that any user that's using HA will need to understand that
>> some machines are running state servers, and when things fail, they
>> will need to manage those machines individually (for example by calling
>> destroy-machine).
>>
>> I also think that the solution to c) is limiting, because there is
>> actually no such thing as a "state server" - we have at least three
>> independently scalable juju components (the database servers (mongodb),
>> the API servers and the environment managers) with different scaling
>> characteristics. I believe that in any sufficiently large environment,
>> the user will not want to scale all of those at the same rate. For example
>> MongoDB will allow at most 12 members of a replica set, but a caching API
>> server could potentially usefully scale up much higher than that. We could
>> add more flags to ensure-ha-state (e.g.--state-server-count) but we then
>> we'd lack the capability to suggest which might be grouped with which.
>>
>> PROPOSAL
>>
>> My suggestion is that we go for a "slightly less magic" approach.
>> that provides the user with the tools to manage
>> their own high availability set up, adding appropriate automation in time.
>>
>> I suggest that we let the user know that machines can run as juju server
>> nodes, and provide them with the capability to *choose* which machines
>> will run as server nodes and which can host units - that is, what *jobs*
>> a machine will run.
>>
>> Here's a possible proposal:
>>
>> We already have an "add-machine" command. We'd add a "--jobs" flag
>> to allow the user to specify the jobs that the new machine(s) will
>> run. Initially we might have just two jobs, "manager" and "unit"
>> - the machine can either host service units, or it can manage the
>> juju environment (including running the state server database),
>> or both. In time we could add finer levels of granularity to allow
>> separate scalability of juju server components, without losing backwards
>> compatibility.
>>
>> If the new machine is marked as a "manager", it would run a mongo
>> replica set peer. This *would* mean that it would be possible to have
>> an even number of mongo peers, with the potential for a split vote
>> if the nodes were partitioned evenly, and resulting database stasis.
>> I don't *think* that would actually be a severe problem in practice.
>> We would make juju status point out the potential problem very clearly,
>> just as it should point out the potential problem if one of an existing
>> odd-sized replica set dies. The potential problems are the same in both
>> cases, and are straightforward for even a relatively naive user to avoid.
>>
>> Thus, juju ensure-ha-state is almost equivalent to:
>>
>> juju add-machine --jobs manager -n 2
>>
>> In my view, this command feels less "magic" than ensure-ha-state - the
>> runtime implication (e.g. cost) of what's going on are easier for the
>> user to understand and it requires no new entities in a user's model of
>> the system.
>>
>> In addition to the new add-machine flag, we'd add a single new command,
>> "juju machine-jobs", which would allow the user to change the jobs
>> associated with an existing machine. That could be a later addition -
>> it's not necessary in the first cut.
>>
>> With these primitives, I *think* the responsibilities of the system and
>> the model to the user become clearer. Looking back to the original
>> user questions:
>>
>> a) The "state manager" status of certain machines in the status is no
>> longer something entirely divorced from user control - it means something
>> in terms of the commands the user is provided with.
>>
>> b) The user already knows about destroy-machine. They can manage broken
>> state manager machines just as they would manage any other broken machine.
>> Destroy-machine would refuse to destroy the any state server machine that
>> would take the currently connected set of mongo peers below a majority.
>>
>> c) We already have add-machine.
>>
>> d) We already have destroy-machine. See c) above.
>>
>> REQUEST FOR COMMENTS
>>
>> If there is broad agreement on the above, then I propose that
>> we start off by implementing ensure-ha-state with as little
>> internal logic as possible - we don't necessarily need to put the
>> transactional
>> logic in to make sure it works without starting extra machines
>> when many people are calling ensure-ha-state concurrently, for example.
>>
>> In fact, ensure-ha-state could probably be written as a thin layer
>> on top of add-machine --jobs.
>>
>> Thoughts?
>>
>
>
> instead of adding more complexity and concepts, it would be ideal if we
> could reuse the primitives we already have. ie juju environments have three
> user exposed services, that users can add-unit / remove-unit etc. they have
> a juju prefix and therefore are omitted by default from status listing.
> That's a much simpler story to document. how do i scale my state server..
> juju add-unit juju-db... my provisioner juju add-unit juju-provisioner.
>
> -k
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
