Michael Han created ZOOKEEPER-3948:
--------------------------------------
Summary: Introduce a deterministic runtime behavior injection
framework for ZooKeeperServer testing
Key: ZOOKEEPER-3948
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3948
Project: ZooKeeper
Issue Type: New Feature
Components: server, tests
Reporter: Michael Han
Assignee: Michael Han
We'd like to understand how applications built on top of ZooKeeper behave under
various faulty conditions, which is important to build resilient end to end
solutions and avoid ZooKeeper being single point of failure. We'd also like to
achieve this in both unit tests (in process) and integration tests (in and out
of process). Traditional methods of using external fault injection mechanisms
are non deterministic and requires non trivial set up and hard to integrate
with unit tests, so here we introduce the ZooKeeperController service which
solves both.
The basic idea here is to create a controllable ZooKeeperServer which accepts
various control commands (such as - delay request, drop request, eat request,
expire session, shutdown, trigger leader election, and so on), and reacting
based on incoming commands. The controllable server and production server share
the same underlying machineries (quorum peers, ZooKeeper server, etc) but code
paths are separate, thus this feature has no production impact.
This controller system is currently composed of following pieces:
* CommandClient: a convenient HTTP client to send control commands to
controller service.
* CommandListener: an embed HTTP server listening incoming commands and
dispatch to controller service.
* Controller Service: the service that's responsible to create controllable ZK
server and the controller.
* ZooKeeperServerController: the controller that changes the behavior of ZK
server runtime.
* Controllable Cnx / Factory: controllable connection that accept behavior
change requests.
In future more control commands and controllable components can be added on top
of this framework.
This can be used in either unit tests / integration tests as an in process
embedded controllable ZooKeeper server, or as an out of process stand alone
controllable ZooKeeper process.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
