[
https://issues.apache.org/jira/browse/KAFKA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905510#comment-16905510
]
Raman Gupta edited comment on KAFKA-8789 at 8/12/19 7:23 PM:
-------------------------------------------------------------
I'm reopening this. The problem does not appear to be with the SR tooling at
all, but rather that the console consumer for the Kafka version included in
Confluent 5.3.0 is a lot slower than the console consumer in 5.0.3. Using a
timeout of 15s is consistently enough to read all messages on the topic in
5.0.3 but has to be at least 60s in 5.3.0, against the same brokers and with
the same parameters.
Also interesting is that the total time for the command to run is pretty much
the same:
{code:java}
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps{code}
so perhaps the behavior of the `--timeout-ms` parameter has changed?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout.
was (Author: rocketraman):
I'm reopening this. The problem does not appear to be with the SR tooling at
all, but rather that the console consumer for the Kafka version included in
Confluent 5.3.0 is a lot slower than the console consumer in 5.0.3. Using a
timeout of 15s is consistently enough to read all messages on the topic in
5.0.3 but has to be at least 60s in 5.3.0, against the same brokers and with
the same parameters.
Ok more information here... the problem appears to be that the 5.3.0 client
needs a much bigger timeout to work consistently i.e. 60s instead of 15s. At
15s, the 5.0.3 client works consistently, but the 5.3.0 client times out every
time. The SR is simply never called because the consumer never receives any
messages within the timeout. So this may be a performance regression with Kafka
and is not related to the SR, or a change in the behavior of the `--timeout-ms`
command.
Also interesting is that the total time for the command to run is pretty much
the same:
{code:java}
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
2.60user 0.22system 0:32.15elapsed 8%CPU (0avgtext+0avgdata 145764maxresident)k
0inputs+0outputs (0major+33989minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1
[...]
Processed a total of 1 messages
3.09user 0.28system 0:32.43elapsed 10%CPU (0avgtext+0avgdata 176440maxresident)k
0inputs+0outputs (0major+40773minor)pagefaults 0swaps
confluent-5.0.3 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[...]
Processed a total of 1 messages
2.58user 0.24system 0:32.29elapsed 8%CPU (0avgtext+0avgdata 144780maxresident)k
0inputs+0outputs (0major+33562minor)pagefaults 0swaps
confluent-5.3.0 $ time kafka-avro-console-consumer <...> --from-beginning
--max-messages 1 --timeout-ms 15000
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
[2019-08-12 15:19:51,214] ERROR Error processing message, terminating consumer
process: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
2.09user 0.17system 0:31.47elapsed 7%CPU (0avgtext+0avgdata 149300maxresident)k
0inputs+8outputs (0major+33949minor)pagefaults 0swaps{code}
so perhaps the behavior of the `--timeout-ms` parameter has changed?
As an aside, the only reason I need the timeout here is because this command is
part of a unix pipeline, and I need it to exit when there are no more messages
to read. Unfortunately, there doesn't appear to be any way to do that except to
set a timeout.
> kafka-console-consumer performance regression
> ---------------------------------------------
>
> Key: KAFKA-8789
> URL: https://issues.apache.org/jira/browse/KAFKA-8789
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Affects Versions: 2.3.0
> Reporter: Raman Gupta
> Priority: Major
>
> I have a topic with about 20,000 events in it. When I run the following tools
> command using Kafka 2.
> bin/kafka-avro-console-consumer \
> --bootstrap-server $KAFKA --property schema.registry.url=$SCHEMAREGISTRY \
> --topic $TOPICPREFIX-user-clickstream-events-ui-v2 \
> --from-beginning --max-messages 100 \
> --isolation-level read_committed --skip-message-on-error \
> --timeout-ms 15000
> I get 100 messages as expected.
> However, when running the exact same command using Kafka 2.3.0 I get
> org.apache.kafka.common.errors.TimeoutException, and 0 messages processed.
> The version of Kafka on the server is 2.3.0.
> NOTE: I am using the Confluent distribution of Kafka for the client side
> tools, specifically Confluent 5.0.3 and Confluent 5.3.0. I can certainly try
> to replicate with a vanilla Kafka if necessary.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
