Bankim Bhavsar created KUDU-3305:
------------------------------------
Summary: Dynamic client configuration for Kudu masters
Key: KUDU-3305
URL: https://issues.apache.org/jira/browse/KUDU-3305
Project: Kudu
Issue Type: Improvement
Components: client, master
Reporter: Bankim Bhavsar
Client configuration requires all masters to be specified while building the
kudu client and connecting to the cluster.
With KUDU-2200 the requirement was relaxed wherein as long as the leader master
is included in the list, connecting to the cluster will be successful. Comment
on the JIRA recommends against relaxing the requirement to specify any master
as that master could fail.
With the common case of configuring multiple masters from a single master
configuration (1 -> 3), it’s possible that one of the newly added masters
becomes the leader. In such a case, if the client configuration is not updated
the requests to the cluster will start failing as per current implementation.
Relaxing the requirement that any master can be specified in the client
configuration will not require the client configuration to be updated
immediately as long as any of the specified masters continues to be part of
multi-master config even as a follower. For the case when all the masters
specified in client configuration fail despite having at least one healthy
master, the requests to the cluster will start failing which is no different
than the current implementation.
As long any master, irrespective of whether it’s a leader, specified in the
client configuration is part of the current multi-master configuration the
cluster will continue to function.
Adding comment from [~granthenke] delving into implementation
“Currently we require that all masters are listed in client configurations.
Instead we could require at least one is listed and as a part of the first step
in connecting a client send a GetMasterAddresses request to discover the
others. This can be done in a round robin fashion until a live master is found
and reports all the configured masters. In this way users can migrate their
master/masters and then slowly adjust the master list over time. This could
also allow them to use a loadbalancer/proxy to resolve one of the master
addresses and then get the real addresses in the first response.
In fact even tablet servers could be used to bootstrap the list of masters.
Optionally using something like a cluster id could be helpful so users can be
confident the bootstrap to the correct cluster.”
>From an implementation perspective, if the leader is not part of the client
>configuration then the client should run the loop once again to determine the
>leader. Once against the list of configured masters as specified in the client
>configuration and if leader is not part of the client configuration then next
>against list of masters as returned by the follower masters in the client
>configuration. In case of discrepancy between masters returned by configured
>master(s), it’s safer to fail determining the leader.
Relaxing the master leader requirement in client configuration will allow
clients to slowly migrate to the updated list of master addresses when new
masters are added.
Another enhancement, suggested by Grant, would be addition of cluster id to
ensure masters are added/removed from the correct cluster and prevents
accidental misconfiguration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
