[
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417880#comment-15417880
]
Elias Levy commented on FLINK-4035:
-----------------------------------
FWIW I generated a flink-connector-kafka-0.9_2.11-1.1.1.jar that uses
kaka-clients 0.10.0.1 (it required hacking around some issues in one of the
tests which I largely ignore). I've tested it in a Flink 1.1.1 cluster against
a 0.10.0.1 Kafka cluster without any issues. Making use of the 0.10.0.1
clients dropped the CPU usage on the Kafka brokers from 100% to 2%, as
previously the broker had to transcode the messages from the 0.10 format to the
0.9 format, whereas with the 0.10 client it can make use of zero copy from disk
to the socket.
It is really too bad that the Kafka clients are not backwards compatible with
older brokers. If they were, that would obviate the need to support multiple
Kafka client version concurrently in Flink and similar system. We'd just have
to keep up with the latest version of the client.
> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---------------------------------------------------
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.0.3
> Reporter: Elias Levy
> Assignee: Robert Metzger
> Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.
> Published messages now include timestamps and compressed messages now include
> relative offsets. As it is now, brokers must decompress publisher compressed
> messages, assign offset to them, and recompress them, which is wasteful and
> makes it less likely that compression will be used at all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
