[
https://issues.apache.org/jira/browse/KAFKA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094983#comment-15094983
]
Guozhang Wang commented on KAFKA-2843:
--------------------------------------
I looked through the code again: HW is stored as a volatile variable on the
server side, AND read operation is not lock protected. So when the fetch
response data is returned, its HW may be updated by another concurrent thread
at the same time.
So in general HW is not a perfect safe check for messageSize > fetchSize, while
it was mainly designed for tracking consuming latency (as HW - fetchedOffset).
For you to monitor if a message's size is too large and hence hanging the
consumer fetching process, in practice (e.g. at LI we used to do this) you can
1) just monitor the fetching latency as described above, and set an alert when
it increased beyond a threshold, and 2) in possible, set the server side config
"message.max.bytes" to the same as the consumer fetch size, so that too-large
messages will be rejected from the server, and hence the producers will be
notified when this happens.
> when consumer got empty messageset, fetchResponse.highWatermark !=
> current_offset?
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-2843
> URL: https://issues.apache.org/jira/browse/KAFKA-2843
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
> Affects Versions: 0.8.2.1
> Reporter: netcafe
>
> I use simple consumer fetch message from brokers (fetchSize >
> messageSize)，when consumer got empty messageSet，e.g ：
> val offset = nextOffset
> val request = buildRequest(offset)
> val response = consumer.fetch(request)
> val msgSet = fetchResponse.messageSet(topic, partition)
>
> if (msgSet.isEmpty) {
> val hwOffset = fetchResponse.highWatermark(topic, partition)
>
> if (offset == hwOffset) {
> // ok, doSomething...
> } else {
> // in our scene, i found highWatermark may not equals current offset
> ，but we did not reproduced it.
> // Is this case could happen ? if could, why ?
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
