[
https://issues.apache.org/jira/browse/KAFKA-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437933#comment-13437933
]
Jun Rao commented on KAFKA-376:
-------------------------------
Prashanth, thanks for the updates.
1) We have a separate jira kafka-420 to track the issue of maintaining HW with
just 1 replica. It probably needs a bit more work than what's in your patch.
First of all, we should increase HW as long as ISR (not assigned replica) has
only 1 replica. Second, we need to check if we can increase HW in at least two
other places (a) when a replica becomes the leader, (b) when ISR shrinks to
size 1.
2) A new leader should never truncate its log. Only a follower truncates its
log. This is because the new leader is guaranteed to have all committed data in
its log and truncating any data from its log may risk losing a committed
message.
3) Yes, we should fix the getOffset api. We will need to distinguish requests
from a follower and a regular client, so that we can return either LEO or HW
accordingly. We can add a replicaId to the getOffsetRequest like FetchRequest.
4) There are a couple other jiras that complicate things a bit. (a) In
kafka-461, we are trying to get rid of the support of magic byte 0 in Message,
which will end up remove BackwardsCompatibilityTest. However, if
BackwardsCompatibilityTest is the only test that fails for this jira, that
means we don't have enough unit test coverage for this jira and kafka-420. So,
we will need to add a new unit test. (b) kafka-351 is doing some refactoring of
ReplicaManager, which will change a bit how KafkaApis interacts with
ReplicaManager.
Ideally, we probably should fix things in the following order. (a) get
kafka-351 (I hope to check in mid this week) in first so that we can use the
cleaner api in ReplicaManager; (b) fix kafka-420 and add a new unit test; (c)
fix this jira and patch OffsetRequest. Does this make sense to you? We can
probably have someone else work on kafka-420 if you don't have the time.
> expose different data to fetch requests from the follower replicas and
> consumer clients
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-376
> URL: https://issues.apache.org/jira/browse/KAFKA-376
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Labels: bugs
> Attachments: KAFKA-376-DRAFT.patch, KAFKA-376-v1.patch,
> KAFKA-376-v2.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, the broker always uses highwatermark to calculate the available
> bytes to a fetch request, no matter where the request is from. Instead, we
> should use highwatermark for requests coming from real consumer clients and
> use logendoffset for requests coming from follower replicas.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
