Julien Fabre created KAFKA-7230:
-----------------------------------
Summary: Empty Record created when producer failed due to
RecordTooLargeException
Key: KAFKA-7230
URL: https://issues.apache.org/jira/browse/KAFKA-7230
Project: Kafka
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Julien Fabre
When a producer try to produce a RecordBatch which is bigger than the
message.max.bytes value, it fails with the error
{code:java}org.apache.kafka.common.errors.RecordTooLargeException{code}
but an empty Record gets created.
While hitting the RecordTooLargeException is expected, I was not expecting
seeing a new offset with an empty Record in the Topic.
Is that a problem with Kafka or should the consumer handle this case ?
Test setup :
- Kafka 2.11-1.1.0
- The producer is written in Go, using a
[SyncProducer|https://godoc.org/github.com/Shopify/sarama#SyncProducer] from
the Sarama library.
- The consumer is kafkacat version 1.3.1-13-ga6b599
Debugs logs from Kafka :
{code}
[2018-08-01 17:21:11,201] DEBUG Accepted connection from /172.17.0.1:33718 on
/172.17.0.3:9092 and assigned it to processor 1, sendBufferSize
[actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
[102400|102400] (kafka.network.Acceptor)
[2018-08-01 17:21:11,201] DEBUG Processor 1 listening to new connection from
/172.17.0.1:33718 (kafka.network.Processor)
[2018-08-01 17:21:11,203] DEBUG [ReplicaManager broker=1001] Request key
events-0 unblocked 0 fetch requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,203] DEBUG [Partition events-0 broker=1001] High watermark
updated to 2 [0 : 136] (kafka.cluster.Partition)
[2018-08-01 17:21:11,203] DEBUG Sessionless fetch context returning 1
partition(s) (kafka.server.SessionlessFetchContext)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key
events-0 unblocked 1 fetch requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key
events-0 unblocked 0 producer requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key
events-0 unblocked 0 DeleteRecordsRequest. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Produce to local
log in 2 ms (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,205] DEBUG Created a new full FetchContext with 1
partition(s). Will not try to create a new session. (kafka.server.FetchManager)
[2018-08-01 17:21:11,210] DEBUG [ReplicaManager broker=1001] Produce to local
log in 0 ms (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,210] DEBUG [KafkaApi-1001] Produce request with
correlation id 1 from client sarama on partition events-0 failed due to
org.apache.kafka.common.errors.RecordTooLargeException (kafka.server.KafkaApis)
{code}
Debug logs from kafkacat :
{code}
%7|1533144071.204|SEND|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Sent FetchRequest (v4, 70 bytes @ 0, CorrId 89)
%7|1533144071.309|RECV|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Received FetchResponse (v4, 50 bytes, CorrId 89, rtt
104.62ms)
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Topic events [0] MessageSet size 0, error "Success",
MaxOffset 2, Ver 2/2
%7|1533144071.309|BACKOFF|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: events [0]: Fetch backoff for 500ms: Broker: No more
messages
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Topic events [0] in state active at offset 0 (1/100000
msgs, 0/1000000 kb queued, opv 2) is not fetchable: fetch backed off
%7|1533144071.309|FETCHADD|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Removed events [0] from fetch list (0 entries, opv 2)
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]:
localhost:9092/1001: Fetch backoff for 499ms
% Reached end of topic events [0] at offset 2
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
