[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180332#comment-15180332
]
James Cheng commented on KAFKA-3334:
------------------------------------
I believe part of the explanation for this is the value of auto.offset.reset,
which defaults to largest. When the consumer is starting up, it tries to access
the topic and fails. By the time it retries, the topic has been created but
also has data in it, so the consumer tries to connect at "largest", which is
the offset of the most recent bit of data, and so it never receives it.
I believe you may be able to avoid/mitigate this by using
auto.offset.reset=smallest.
Btw, this has impact on mirrormaker. If you have a mirror whitelist and mirror
new topics as they are created (whether manually or automatically), and if
mirrormaker has auto.offset.reset=largest, then mirrormaker may miss messages
when new topics are created.
> First message on new topic not actually being sent, no exception thrown
> -----------------------------------------------------------------------
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0
> Environment: Linux, Java
> Reporter: Aleksandar Stojadinovic
> Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future<RecordMetadata> recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
