[
https://issues.apache.org/jira/browse/KAFKA-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542550#comment-13542550
]
Neha Narkhede commented on KAFKA-513:
-------------------------------------
Thanks for patch v1. Overall looks pretty good, a few review suggestions and
questions -
1. Partition.scala
1.1 The state change log is a timeline of state change requests sent by the
controller and state change requests completed by the broker. On the broker
side, it will be nice to see something like this -
Controller 3 sent the follower state transition for [foo,1] to broker 1
Broker 1 started follower state transition for [foo,1] to leader 2
Broker 1 completed follower state transition for [foo,1] to leader 2
Broker 1 aborted follower state transition for [foo,1] to leader 2
1.2 It will be good to standardize on our logging style for partition. There
are some log statements that use [topic, partition], others that use
[topic,partition] and some more that use "topic %s partition %d". I personally
prefer [topic,partition] since it is concise and does not have extra white
spaces.
1.3 Given that we already have one log statement on the broker stating start of
the state transition, I think we can get rid of the following from the state
change log
Started to become follower at the request %s
2. PartitionStateMachine
2.1 There are 2 ways I see the log messages from controller being logged. One
is a specialized formatted log that is the state change log. This is used by
the state change log merge tool to create a timeline of state changes for a
particular partition. Another is re-directing log4j statements from
kafka.controller to a separate log4j appender. The purpose would be keeping the
log statements specific to controller from polluting the broker logs since
presumably only Kafka developers or advanced users would be able to make sense
of it. Having said that, I think we can keep the debug statements and anything
that helps troubleshooting out of the specialized state change log since that
will be captured in the controller's log anyway.
2.2 The leader election state change log messages can be something like
Controller 3 started leader election for [foo,1]
Controller 3 completed leader election for [foo,1]
3. PartitionStateChangeLogger
Since this is a thin wrapper over the Logging trait, do we really need this
class at all ? Or, to maintain consistency, we can just use it as a standard
logger similar to the requestLogger in KafkaApis.
4. ReplicaStateMachine
Given that we already have the state change message for OfflineReplica, I think
we don't need the following message about removing it from the ISR
5. log4j.properties
How about calling it just a stateChangeAppender and the log a state_change.log
?
6. PartitionStateChangeLogMerger
6.1 It's unclear what the input to this tool is. It will help to give a more
detailed description to the input options.
6.2 The way I see it, there are at least 3 ways to merge the state change logs
- Filter all the state changes on a partition or all partitions for a topic or
everything. We should probably provide these options as input
6.3 Currently, the merging algorithm requires streaming all the entries from
each broker's state change log in memory. I wonder if this runs a risk of
running out of memory. On one hand, I think there should be fewer state change
log entries, so this approach should work just fine. On the other hand, if
there is a problem, leading to a lot of state change traffic, this tool might
not hold up. One way to mediate this is to basically do a n-way merge from m
input files. The algorithm is something like this -
6.3.1 Read a line (that matches the topic/partition regex if there is one) from
every input file in a priority queue
6.3.2 Take the line from the file with the earliest date and add it to an
output buffer
6.3.3 Add another line from the file in the earlier step in the priority queue
6.3.4 Flush the output buffer every n entries
This approach will bound the memory required by this tool to the amount of
memory required to store roughly n log entries
> Add state change log to Kafka brokers
> -------------------------------------
>
> Key: KAFKA-513
> URL: https://issues.apache.org/jira/browse/KAFKA-513
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Priority: Blocker
> Labels: replication, tools
> Fix For: 0.8
>
> Attachments: kafka-513-v1.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Once KAFKA-499 is checked in, every controller to broker communication can be
> modelled as a state change for one or more partitions. Every state change
> request will carry the controller epoch. If there is a problem with the state
> of some partitions, it will be good to have a tool that can create a timeline
> of requested and completed state changes. This will require each broker to
> output a state change log that has entries like
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() for
> partition [foo, 0] from controller 2, epoch 1
> On controller, this will look like -
> [2012-09-10 10:06:17,198] controller 2, epoch 1, initiated state change
> request LeaderAndIsr() for partition [foo, 0]
> We need a tool that can collect the state change log from all brokers and
> create a per-partition timeline of state changes -
> [foo, 0]
> [2012-09-10 10:06:17,198] controller 2, epoch 1 initiated state change
> request LeaderAndIsr()
> [2012-09-10 10:06:17,280] broker 1 received request LeaderAndIsr() from
> controller 2, epoch 1
> [2012-09-10 10:06:17,350] broker 1 completed request LeaderAndIsr() from
> controller 2, epoch 1
> This JIRA involves adding the state change log to each broker and adding the
> tool to create the timeline
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
