[jira] [Commented] (KAFKA-5390) First records in batch rejected but others accepted when rolling log

[Magnus Reftel (JIRA)]

    [ 
https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038832#comment-16038832
 ] 

Magnus Reftel commented on KAFKA-5390:
--------------------------------------

Thanks for the quick reply! I tried switching 
{{records.forEach(producer::send);}} to

{code}
        records.forEach(r -> producer.send(r, (m, e) -> {
            System.out.printf("Send result for %s: %s\n", r.key(), m == null ? 
"null" : "\"" + m + "\"");
            if (e != null) {
                e.printStackTrace();
            }
        }));
{code}

This produces the following output:

{noformat}
Send result for 0: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 1: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 2: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 3: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 4: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 5: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 6: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 7: null
org.apache.kafka.common.errors.RecordBatchTooLargeException: The request 
included message batch larger than the configured segment size on the server.
Send result for 8: "t-0@0"
{noformat}

The testcase still fails for the same reason as before.

> First records in batch rejected but others accepted when rolling log
> --------------------------------------------------------------------
>
>                 Key: KAFKA-5390
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5390
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.2.1
>            Reporter: Magnus Reftel
>         Attachments: kafka-sequence.tar.gz
>
>
> When sending a sequence of records in a batch right when the broker needs to 
> roll a new segment, it's possible for the first few records to fail, while 
> other records in the batch are accepted. If records have dependencies on 
> earlier records, e.g. in the case of a sequence of events in an event-sourced 
> system, then a producer cannot use the batching functionality, since it then 
> risks consumers receiving a record without first receiving the records it 
> depends on.
> See attached testcase (kafka-sequence.tar.gz).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)