[ https://issues.apache.org/jira/browse/KAFKA-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038969#comment-16038969 ]
Ismael Juma commented on KAFKA-5390: ------------------------------------ If you want to do manual batching, you can simply increase `batch.size` to a large number and use `flush` whenever you want to create a batch. Be aware that you need to ensure that the batch size is not larger than `max.request.size` (which you can increase as well, of course). You also need to configure the brokers appropriately: max.message.size, log.segment.bytes, replica.fetch.max.bytes, replica.fetch.response.max.bytes. And the consumer: fetch.max.bytes, max.partition.fetch.bytes. > First records in batch rejected but others accepted when rolling log > -------------------------------------------------------------------- > > Key: KAFKA-5390 > URL: https://issues.apache.org/jira/browse/KAFKA-5390 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.10.2.1 > Reporter: Magnus Reftel > Attachments: kafka-sequence-2.tar.gz, kafka-sequence.tar.gz > > > When sending a sequence of records in a batch right when the broker needs to > roll a new segment, it's possible for the first few records to fail, while > other records in the batch are accepted. If records have dependencies on > earlier records, e.g. in the case of a sequence of events in an event-sourced > system, then a producer cannot use the batching functionality, since it then > risks consumers receiving a record without first receiving the records it > depends on. > See attached testcase (kafka-sequence.tar.gz). -- This message was sent by Atlassian JIRA (v6.3.15#6346)