[
https://issues.apache.org/jira/browse/FLINK-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597662#comment-16597662
]
ASF GitHub Bot commented on FLINK-7964:
---------------------------------------
eliaslevy commented on issue #6577: [FLINK-7964] Add Apache Kafka 1.0/1.1
connectors
URL: https://github.com/apache/flink/pull/6577#issuecomment-417387441
@aljoscha
* [KIP-35 - Retrieving protocol
version](https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version)
* [KIP-97: Improved Kafka Client RPC Compatibility
Policy](https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy)
> This KIP suggests revising the Kafka Java client compatibility policy so
that it is two-way, rather than one-way. Not only should older brokers support
newer clients, but newer clients should support older brokers.
> In some cases, older brokers will be unable to perform a feature supported
by newer clients. For example, retrieving message offsets from a message
timestamp (KIP-79) is not supported in versions before 0.10.1. In these cases,
the new client will throw an ObsoleteBrokerVersion exception when attempting to
perform the operation. This will let clients know that the reason why the
operation cannot be performed is because the broker version in use does not
support it. At the same time, clients which do not attempt to use the new
operations will proceed as before.
* [Compatibility
Matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix)
Basically, any KIP-35 client can talk to a Kafka broker >= 0.10.0, which for
the Java client
[means](https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version)
versions >= 0.10.2.
@pnowojski I am suggesting that, except possibly for 0.8 and 0.9 clients,
there is only a need for a single Kafka connector based on the latest Kafka
client (2.0.0 as of now). There is no need for separate 2.0.0, 1.x.y,
0.11.0.x, and 0.10.x.y connectors. A connector based on 2.0.0 can talk to all
those brokers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Apache Kafka 1.0/1.1 connectors
> -----------------------------------
>
> Key: FLINK-7964
> URL: https://issues.apache.org/jira/browse/FLINK-7964
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Affects Versions: 1.4.0
> Reporter: Hai Zhou
> Assignee: vinoyang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Kafka 1.0.0 is no mere bump of the version number. The Apache Kafka Project
> Management Committee has packed a number of valuable enhancements into the
> release. Here is a summary of a few of them:
> * Since its introduction in version 0.10, the Streams API has become hugely
> popular among Kafka users, including the likes of Pinterest, Rabobank,
> Zalando, and The New York Times. In 1.0, the the API continues to evolve at a
> healthy pace. To begin with, the builder API has been improved (KIP-120). A
> new API has been added to expose the state of active tasks at runtime
> (KIP-130). The new cogroup API makes it much easier to deal with partitioned
> aggregates with fewer StateStores and fewer moving parts in your code
> (KIP-150). Debuggability gets easier with enhancements to the print() and
> writeAsText() methods (KIP-160). And if that’s not enough, check out KIP-138
> and KIP-161 too. For more on streams, check out the Apache Kafka Streams
> documentation, including some helpful new tutorial videos.
> * Operating Kafka at scale requires that the system remain observable, and to
> make that easier, we’ve made a number of improvements to metrics. These are
> too many to summarize without becoming tedious, but Connect metrics have been
> significantly improved (KIP-196), a litany of new health check metrics are
> now exposed (KIP-188), and we now have a global topic and partition count
> (KIP-168). Check out KIP-164 and KIP-187 for even more.
> * We now support Java 9, leading, among other things, to significantly faster
> TLS and CRC32C implementations. Over-the-wire encryption will be faster now,
> which will keep Kafka fast and compute costs low when encryption is enabled.
> * In keeping with the security theme, KIP-152 cleans up the error handling on
> Simple Authentication Security Layer (SASL) authentication attempts.
> Previously, some authentication error conditions were indistinguishable from
> broker failures and were not logged in a clear way. This is cleaner now.
> * Kafka can now tolerate disk failures better. Historically, JBOD storage
> configurations have not been recommended, but the architecture has
> nevertheless been tempting: after all, why not rely on Kafka’s own
> replication mechanism to protect against storage failure rather than using
> RAID? With KIP-112, Kafka now handles disk failure more gracefully. A single
> disk failure in a JBOD broker will not bring the entire broker down; rather,
> the broker will continue serving any log files that remain on functioning
> disks.
> * Since release 0.11.0, the idempotent producer (which is the producer used
> in the presence of a transaction, which of course is the producer we use for
> exactly-once processing) required max.in.flight.requests.per.connection to be
> equal to one. As anyone who has written or tested a wire protocol can attest,
> this put an upper bound on throughput. Thanks to KAFKA-5949, this can now be
> as large as five, relaxing the throughput constraint quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
