[
https://issues.apache.org/jira/browse/KAFKA-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-499:
--------------------------------
Attachment: kafka-499-v1.patch
This patch refactors (almost rewrites) the controller. The controller has the
following components -
1. A partition state machine.
2. A replica state machine
3. Controller channel manager for communication with the brokers.
1. Partition state machine
This represents the state machine for partitions. It defines the states that a
partition can be in, and
transitions to move the partition to some legal state. The different states
that a partition can be in are -
1. NonExistentPartition: This state indicates that the partition was either
never created or was created and then
deleted. Valid previous state, if one exists, is OfflinePartition
2. NewPartition : After creation, the partition is in the NewPartition
state. In this state, the partition should have replicas assigned to it, but no
leader/isr yet. Valid previous states are NonExistentPartition
3. OnlinePartition : Once a leader is elected for a partition, it is in
the OnlinePartition state. Valid previous states are
NewPartition/OfflinePartition
4. OfflinePartition : If, after successful leader election, the leader for
partition dies, then the partition moves to the OfflinePartition state. Valid
previous states are NewPartition/OnlinePartition
2. Replica state machine
This class represents the state machine for replicas. It defines the states
that a replica can be in, and
transitions to move the replica to another legal state. The different states
that a replica can be in are -
1. OnlineReplica : Once a replica is started, it is in this state. Valid
previous state are OnlineReplica or OfflineReplica
2. OfflineReplica : If a replica dies, it moves to this state. Valid
previous state is OnlineReplica.
There might be some changes to this state machine after KAFKA-42 is checked in.
For now, these are the relevant states that a replica can be in.
3. Controller channel manager
Everything related to communication between controller and broker is moved
here. This includes the request channels to the brokers, the request send
thread, and the request batching mechanism. Since the request batching
mechanism is new, I'll explain it here -
ControllerBrokerRequestBatch
3.1. All the state changes initiated by the controller requires batching of
requests per broker. Previously, the code to achieve this was copy pasted in
several state changes APIs. I moved it into one class that allows the
controller to -
3.1.1. newBatch() - start a new request batch when a set of state changes need
to be initiated
3.1.2 sendRequestsToBrokers() - collate the state changes per broker and send
them
4. Testing
While I was testing the controller code, I found that most of our existing
unit tests already cover most of the corner cases. For example, LogRecoveryTest
and LeaderElectionTest. However, I guess the controller needs more system
testing where we have multiple topics and partitions and it -
4.1 Introduces soft (GC) as well as hard failures to trigger controller
failover. Right now, our tests only cover controller failures (clean shutdown)
4.2 Introduce state changes for multiple partitions in quick succession.
I feel the above test could be cleanly done as a system test. Since this patch
blocks KAFKA-42, KAFKA-43 and some other controller related JIRAs, I thought I
can push this to a separate patch or under KAFKA-42. If people feel it is
important to add it as part of this patch, let me know, I can get it in.
5. I wanted to include a state change diagram to make it easier to understand
both the partitions and replica state machines. I think I will add that to the
Internal Design wiki soon.
> Refactor controller state machine
> ----------------------------------
>
> Key: KAFKA-499
> URL: https://issues.apache.org/jira/browse/KAFKA-499
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: optimization
> Attachments: kafka-499-v1.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Currently, the controller logic is very procedural and is similar to
> KafkaZookeeper. Controller should have a well defined state machine with
> states and transitions. This will make it easier to understand and maintain
> the controller code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
