[
https://issues.apache.org/jira/browse/KAFKA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206723#comment-15206723
]
Jun Rao commented on KAFKA-3442:
--------------------------------
There is another tricky issue to think through. If a client sends a fetch
request with a max.partition.fetch.bytes smaller than the message size in a
topic/partition, before 0.10.0, the broker will send a fetch response with a
partition payload of exactly max.partition.fetch.bytes bytes. When the client
iterates the payload, it will realize that it can't iterate a single complete
message and yet the payload size is > 0, it will then just error out with
RecordTooLargeException since it can't make progress w/o a larger
max.partition.fetch.bytes bytes. The current patch instead will send an empty
payload to the client. Then, the client doesn't know that it actually gets
stuck. So, when doing a down conversion, we need to somehow convert a partial
message as well if the first message size is larger than
max.partition.fetch.bytes.
[~dana.powers], [~edenhill], how did you handle this case in the non-java
client? Do you just do the same check as in the java client, i.e., if can't
iterate a single complete message in the payload from a partition in the fetch
response and yet the payload size is > 0, just error out with
RecordTooLargeException? The 0.10.0 broker may need to down convert messages to
v0 format when serving fetch request from pre 0.10.0 clients. It just that
after the conversion, the partial message returned may not be exactly of size
max.partition.fetch.bytes.
> FetchResponse size exceeds max.partition.fetch.bytes
> ----------------------------------------------------
>
> Key: KAFKA-3442
> URL: https://issues.apache.org/jira/browse/KAFKA-3442
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.0.0
> Reporter: Dana Powers
> Assignee: Jiangjie Qin
> Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Produce 10000 byte message to topic foobar
> Fetch foobar w/ max.partition.fetch.bytes=1024
> Test expects to receive a truncated message (~1024 bytes). 0.8 and 0.9 pass
> this test, but 0.10 FetchResponse has full message, exceeding the max
> specified in the FetchRequest.
> I tested with v0 and v1 apis, both fail. Have not tested w/ v2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
