[
https://issues.apache.org/jira/browse/ZOOKEEPER-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566886#comment-13566886
]
Marshall McMullen commented on ZOOKEEPER-107:
---------------------------------------------
We just wanted to give some support for this patch.
It has been extremely helpful to our Zookeeper deployment strategy. We
(www.solidfire.com) use Zookeeper as our database for a distributed scale-out
block storage system. We deploy fully automated clusters from 3 to 25+ nodes
that must be able to self heal ensemble issues after inevitable node failures.
We maintain an ensemble of 3 or 5 nodes, so if one of these ensemble nodes
fails we require another node in the cluster to assume its duties as quickly as
possible. Prior to this patch, the only way to reconfigure the ensemble was to
do a typical "rolling restart" of the Zookeeper process on each of the nodes
after modifying the local configuration file, but manually adjusting the
ensemble wasn't an option. Automating this was a challenge though since
Zookeeper also contained our authoritative node list (chicken and egg), which
meant that cascading failures could easily lead to inconsistencies in
configuration files and then the ensemble could get easily confused.
Additionally, the re-connection events caused by restarting servers complicated
the centralized logic that was coordinating the reconfiguration.
This is where Zookeeper-107 has been a life saver. We integrated this patch
into our production version of Zookeeper-3.5.0 about a year ago and have been
running with it since. The simplicity of the interface and the reliability it
has shown have made handling ensemble node failures significantly easier. We
exclusively use the c-client and have not had any problems with it. Currently
we only use the "full reconfiguration" functionality rather than the
"incremental" reconfiguration. In integrating this functionality with our
coordination code, all of the problems we ran into were *ours* and the patch
showed solid resilience to the many error scenarios we've thrown at it.
Furthermore, Alex was super helpful and responsive in helping us integrate this
into our environment. Eventually we want to write up a small white paper on
exactly how we manage our ensemble, but we don't have the time right now.
Just to give some perspective on the amount of stress this patch has gotten in
our codebase, here are some stats. The reconfiguration code has ran 10,000+
times within our integration testing and product qualification environments.
The sorts of tests we have in our integration testing environment are heavily
focused on reconfiguring the ensemble while there are concurrent operations in
flight simultaneously with the reconfiguration operation. We've never seen an
operation fail or get missed in any way during the reconfiguration. We also
have an extensive set of tests which essentially create a three node ensemble
and then add two more nodes and grow the ensemble to 5. The test has two
different Zookeeper clients -- one connected to the original set of 3 and one
connected to the new set of 2 that is joining the ensemble. While we are
reconfiguring the ensemble to grow from 3 to 5 we hammer the original 3 nodes
with write operations. Once the reconfiguration is complete, we validate that
the new 2 nodes see all the operations submitted to the original 3 nodes. We've
never seen any problems with any of our tests!
Given its stability in house, we are now actively deploying Zookeeper-3.5.0
with this patch to several customers running production environments (and many
more running test environments), which combined with our internal test
environment means we have this software running on 200+ nodes in 25-50+
clusters at any one time. Our website shows the importance these customers
place on our software, so you can be sure we've tested it extensively in
hundreds of different reconfiguration scenarios, and we haven't had to file a
single jira yet.
> Allow dynamic changes to server cluster membership
> --------------------------------------------------
>
> Key: ZOOKEEPER-107
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-107
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Patrick Hunt
> Assignee: Alexander Shraer
> Fix For: 3.5.0
>
> Attachments: SimpleAddition.rtf, zkreconfig-usenixatc-final.pdf,
> ZOOKEEPER-107-14-Jan.patch, ZOOKEEPER-107-14-Oct.patch,
> ZOOKEEPER-107-15-Oct.patch, ZOOKEEPER-107-15-Oct-ver1.patch,
> ZOOKEEPER-107-15-Oct-ver2.patch, ZOOKEEPER-107-15-Oct-ver3.patch,
> ZOOKEEPER-107-16-Jan.patch, ZOOKEEPER-107-17-Jan.patch,
> ZOOKEEPER-107-18-Jan.patch, ZOOKEEPER-107-18-Jan-ver2.patch,
> ZOOKEEPER-107-1-Mar.patch, ZOOKEEPER-107-20-Jan.patch,
> ZOOKEEPER-107-20-July.patch, ZOOKEEPER-107-21-July.patch,
> ZOOKEEPER-107-22-Apr.patch, ZOOKEEPER-107-23-SEP.patch,
> ZOOKEEPER-107-24-Jan.patch, ZOOKEEPER-107-28-Feb.patch,
> ZOOKEEPER-107-28-Feb.patch, ZOOKEEPER-107-28-NOV-ver2.patch,
> ZOOKEEPER-107-29-Feb.patch, ZOOKEEPER-107-3-Oct.patch,
> ZOOKEEPER-107-6-NOV-2.patch, ZOOKEEPER-107-7-NOV.patch,
> ZOOKEEPER-107-7-NOV-ver1.patch, ZOOKEEPER-107-7-NOV-ver2.patch,
> ZOOKEEPER-107-Aug-20.patch, ZOOKEEPER-107-Aug-20-ver1.patch,
> ZOOKEEPER-107-Aug-25.patch, zookeeper-3.4.0.jar, zookeeper-dev-fatjar.jar,
> zookeeper-reconfig-sep11.patch, zookeeper-reconfig-sep12.patch,
> zoo_replicated1.cfg, zoo_replicated1.members
>
>
> Currently cluster membership is statically defined, adding/removing hosts
> to/from the server cluster dynamically needs to be supported.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
