[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220277#comment-15220277
]
Aleksandar Stojadinovic commented on KAFKA-3334:
I agree about overexposing the internals (at least in one document). A mention
in FAQ, or even better, a chapter about auto-created topics should be more like
it. I'm mentioning a separate chapter because I believe it will catch more
attention. If someone is aiming to use the auto-crated feature, they will
probably take a second and throw a look. Also, people Googling "Kafka publish
autocreate" will have that one high in the results.
To potentially discourage the auto-create feature, is it an option to disable
it by default? And when a user searches for it, they can see a word of notice
in the configuration?
I mean, the best thing would be to never have the issue of missing a message,
but I guess that's easier said than done :) .
> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215668#comment-15215668
]
Aleksandar Stojadinovic commented on KAFKA-3334:
I'm carefully following the discussion.
An exception would be my preferable choice, as an end user. However, now I know
what would be the root cause of the exception. If I were a new user (evaluating
Kafka, maybe), and if it is not properly documented that there might be an
exception on the start, after encountering an exception I might go wandering
around why my cluster is not working.
It's also not completely clear to me, is it the producer's responsibility to
always issue a TopicMetadataRequest before issuing a message, which should
supposedly resolve the "silent fail" issue?
In the meanwhile I moved to manual topic on risky places.
> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183597#comment-15183597
]
Aleksandar Stojadinovic commented on KAFKA-3334:
The addition sounds very reasonable. Maybe just add what does "proper" mean.
Larger, shorter? I'm not sure someone without experience would understand.
> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182767#comment-15182767
]
Aleksandar Stojadinovic commented on KAFKA-3334:
Sure :) .
> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180267#comment-15180267
]
Aleksandar Stojadinovic commented on KAFKA-3334:
Understood. To be honest, I could manage manual creation, the fact that
this happens pretty much silently introduces fear if there are not more
situations like this. To be fair I have not seen them until now.
Maybe document this somehow?
> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown
Aleksandar Stojadinovic created KAFKA-3334:
--
Summary: First message on new topic not actually being sent, no
exception thrown
Key: KAFKA-3334
URL: https://issues.apache.org/jira/browse/KAFKA-3334
Project: Kafka
Issue Type: Bug
Affects Versions: 0.9.0.0
Environment: Linux, Java
Reporter: Aleksandar Stojadinovic
Although I've seen this issue pop around the internet in a few forms, I'm not
sure it is yet properly fixed.
When publishing to a new topic, with auto create-enabled, the java client
(0.9.0) shows this WARN message in the log, and the message is not sent
obviously:
org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
In the meantime I see in the console the message that a log for partition is
created. The next messages are patched through normally, but the first one is
never sent. No exception is ever thrown, either by calling get on the future,
or with the async usage, like everything is perfect.
I notice when I leave my application blocked on the get call, in the debugger,
then the message may be processed, but with significant delay. This is
consistent with another issue I found for the python client. Also, if I call
partitionsFor previously, the topic is created and the message is sent. But it
seems silly to call it every time, just to mitigate this issue.
{code}
Future recordMetadataFuture = producer.send(new
ProducerRecord<>(topic, key, file));
RecordMetadata recordMetadata = recordMetadataFuture.get(30,
TimeUnit.SECONDS);
{code}
I hope I'm clear enough.
Related similar (but not same) issues:
https://issues.apache.org/jira/browse/KAFKA-1124
https://github.com/dpkp/kafka-python/issues/150
http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
