[
https://issues.apache.org/jira/browse/KAFKA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905510#comment-16905510
]
Raman Gupta edited comment on KAFKA-8789 at 8/12/19 11:11 PM:
--------------------------------------------------------------
The problem seems to be related to the `--timeout-ms` parameter.
In every case, the total time for the command to run is pretty much the same:
{code:java}
# Confluent 5.0.3 with no timeout
confluent-5.0.3 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
# Confluent 5.3.0 with no timeout
confluent-5.3.0 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
# Confluent 5.0.3 with 15s timeout
confluent-5.0.3 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
# Confluent 5.3.0 with 15s timeout
confluent-5.3.0 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps
# Confluent 5.3.0 with 45s timeout works
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 45000
Processed a total of 1 messages
3.12user 0.27system 0:32.55elapsed 10%CPU (0avgtext+0avgdata 178252maxresident)k
0inputs+0outputs (0major+41263minor)pagefaults 0swaps
{code}
but newer client versions appear to need a longer timeout to work correctly Has
the behavior of the `--timeout-ms` parameter changed in some way?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout, so having the smallest timeout possible that will likely give me
all the messages is desirable.
was (Author: rocketraman):
I'm reopening this. The problem does not appear to be with the SR tooling at
all, but rather that the console consumer for the Kafka version included in
Confluent 5.3.0 is a lot slower than the console consumer in 5.0.3. Using a
timeout of 15s is consistently enough to read all messages on the topic in
5.0.3 but has to be at least 60s in 5.3.0, against the same brokers and with
the same parameters.
Also interesting is that the total time for the command to run is pretty much
the same:
{code:java}
# Confluent 5.0.3 with no timeout
confluent-5.0.3 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
# Confluent 5.3.0 with no timeout
confluent-5.3.0 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
# Confluent 5.0.3 with 15s timeout
confluent-5.0.3 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
# Confluent 5.3.0 with 15s timeout
confluent-5.3.0 $ time kafka-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps
# Confluent 5.3.0 with 45s timeout works
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 45000
Processed a total of 1 messages
3.12user 0.27system 0:32.55elapsed 10%CPU (0avgtext+0avgdata 178252maxresident)k
0inputs+0outputs (0major+41263minor)pagefaults 0swaps
{code}
so perhaps the behavior of the `--timeout-ms` parameter has changed?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout, so having the smallest timeout possible that will likely give me
all the messages is desirable.
> kafka-console-consumer timeout-ms setting behaves incorrectly with older
> client
> -------------------------------------------------------------------------------
>
> Key: KAFKA-8789
> URL: https://issues.apache.org/jira/browse/KAFKA-8789
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Affects Versions: 2.3.0
> Reporter: Raman Gupta
> Priority: Major
>
> I have a topic with about 20,000 events in it, running on a Kafka 2.3.0
> broker. When I run the following tools command using the older Kafka client
> included in Confluent 5.0.3.
> bin/kafka-console-consumer \
> --bootstrap-server $KAFKA \
> --topic x \
> --from-beginning --max-messages 1 \
> --timeout-ms 15000
> I get 1 message as expected.
> However, when running the exact same command using the console consumer
> included with Confluent 5.3.0, I get
> org.apache.kafka.common.errors.TimeoutException, and 0 messages processed.
> NOTE: I am using the Confluent distribution of Kafka for the client side
> tools, specifically Confluent 5.0.3 and Confluent 5.3.0. I can certainly try
> to replicate with a vanilla Kafka if necessary.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
