[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[
https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664953#comment-16664953
]
ASF GitHub Bot commented on FLINK-10599:
pnowojski closed pull request #6889: [FLINK-10599][Documentation] Provide
documentation for the modern kafka connector
URL: https://github.com/apache/flink/pull/6889
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index b9dbd8fe07f..f81080a4b53 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -80,6 +80,14 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
0.11.x
Since 0.11.x Kafka does not support scala 2.10. This connector
supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging";>Kafka
transactional messaging to provide exactly once semantic for the
producer.
+
+flink-connector-kafka_2.11
+1.7.0
+FlinkKafkaConsumer
+FlinkKafkaProducer
+>= 1.0.0
+This Kafka connector attempts to track the latest version of the
Kafka client. The version of the client it uses may change between Flink
releases. Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend
using dedicated flink-connector-kafka-0.11 and link-connector-kafka-0.10
respectively.
+
@@ -88,8 +96,8 @@ Then, import the connector in your maven project:
{% highlight xml %}
org.apache.flink
- flink-connector-kafka-0.8{{ site.scala_version_suffix
}}
- {{site.version }}
+ flink-connector-kafka{{ site.scala_version_suffix }}
+ {{ site.version }}
{% endhighlight %}
@@ -100,9 +108,33 @@ Note that the streaming connectors are currently not part
of the binary distribu
* Follow the instructions from [Kafka's
quickstart](https://kafka.apache.org/documentation.html#quickstart) to download
the code and launch a server (launching a Zookeeper and a Kafka server is
required every time before starting the application).
* If the Kafka and Zookeeper servers are running on a remote machine, then the
`advertised.host.name` setting in the `config/server.properties` file must be
set to the machine's IP address.
+## Kafka 1.0.0+ Connector
+
+Starting with Flink 1.7, there is a new Kafka connector that does not track a
specific Kafka major version. Rather, it tracks the latest version of Kafka at
the time of the Flink release.
+
+If your Kafka broker version is 1.0.0 or newer, you should use this Kafka
connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you
should use the connector corresponding to the broker version.
+
+### Compatibility
+
+The modern Kafka connector is compatible with older and newer Kafka brokers
through the compatibility guarantees of the Kafka client API and broker. The
modern Kafka connector is compatible with broker versions 0.11.0 or later,
depending on the features used. For details on Kafka compatibility, please
refer to the [Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).
+
+### Usage
+
+The use of the modern Kafka connector add a dependency to it:
+
+{% highlight xml %}
+
+ org.apache.flink
+ flink-connector-kafka{{ site.scala_version_suffix }}
+ {{ site.version }}
+
+{% endhighlight %}
+
+Then instantiate the new source (`FlinkKafkaConsumer`) and sink
(`FlinkKafkaProducer`). The API is the backwards compatible with the older
Kafka connectors.
+
## Kafka Consumer
-Flink's Kafka consumer is called `FlinkKafkaConsumer08` (or `09` for Kafka
0.9.0.x versions, etc.). It provides access to one or more Kafka topics.
+Flink's Kafka consumer is called `FlinkKafkaConsumer08` (or 09 for Kafka
0.9.0.x versions, etc. or just `FlinkKafkaConsumer` for Kafka >= 1.0.0
versions). It provides access to one or more Kafka topics.
The constructor accepts the following arguments:
@@ -492,7 +524,7 @@ In the meanwhile, a possible workaround is to send
*heartbeat messages* to all c
## Kafka Producer
-Flink’s Kafka Producer is called `FlinkKafkaProducer011` (or `010` for Kafka
0.10.0.x versions, etc.).
+Flink’s Kafka Producer is called `FlinkKafkaProducer011` (or `010` for Kafka
0.10.0.x versions, etc. or just `FlinkKafkaConsumer` for Kafka >= 1.0.0
versions).
It allows writing a stream of records to one or more Kafka topics.
Example:
@@ -618,13 +650,13 @@ into a Kafka topic.
for more explanation.
- Kafka 0.11
+ Kafka 0.11 and newer
-With Flink's checkpointing enabled, the `FlinkKafk
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663950#comment-16663950 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-433110004 @pnowojski updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663914#comment-16663914 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228218658 ## File path: docs/dev/connectors/kafka.md ## @@ -108,15 +108,15 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. -## Modern Kafka Connector +## Kafka 0.11+ Connector Review comment: Sorry for previous comment, I've just realised that naming it `Kafka 0.11+ Connector` is a bit inconsistent since everywhere else we are claiming support for `>= 1.0.0". So maybe change this section to `## Kafka 1.0.0+ Connector`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663900#comment-16663900 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228219609 ## File path: docs/dev/connectors/kafka.md ## @@ -650,13 +650,13 @@ into a Kafka topic. for more explanation. - Kafka 0.11 + Kafka 0.11 and newer -With Flink's checkpointing enabled, the `FlinkKafkaProducer011` can provide +With Flink's checkpointing enabled, the `FlinkKafkaProducer011` (for modern Kafka connector, it is `FlinkKafkaProducer`.) can provide Review comment: ditto ( FlinkKafkaProducer for Kafka >= 1.0.0 versions). ) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663896#comment-16663896 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228218658 ## File path: docs/dev/connectors/kafka.md ## @@ -108,15 +108,15 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. -## Modern Kafka Connector +## Kafka 0.11+ Connector Review comment: Sorry for previous comment, I just realised that naming it `Kafka 0.11+ Connector` is a bit inconsistent since everywhere else we are claiming support for `>= 1.0.0". So maybe change this section to `## Kafka 1.0.0+ Connector`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663899#comment-16663899 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228217863 ## File path: docs/dev/connectors/kafka.md ## @@ -85,7 +85,7 @@ For most users, the `FlinkKafkaConsumer08` (part of `flink-connector-kafka`) is 1.7.0 FlinkKafkaConsumer FlinkKafkaProducer ->= 1.x +>= 1.0.0 This Kafka connector attempts to track the latest version of the Kafka client. The version of the client it uses may change between Flink releases. Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. Review comment: Replace to: Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. However for Kafka 0.11.x and 0.10.x versions, we recommend using dedicated `flink-connector-kafka-0.11` and `link-connector-kafka-0.10` respectively. ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663901#comment-16663901 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228219298 ## File path: docs/dev/connectors/kafka.md ## @@ -524,7 +524,7 @@ In the meanwhile, a possible workaround is to send *heartbeat messages* to all c ## Kafka Producer -Flink’s Kafka Producer is called `FlinkKafkaProducer011` (or `010` for Kafka 0.10.0.x versions, etc.). +Flink’s Kafka Producer is called `FlinkKafkaProducer011` (or `010` for Kafka 0.10.0.x versions, etc. Note that since modern Kafka connector, it will not carry the version number of Kafka.). Review comment: ditto? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663897#comment-16663897 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228219170 ## File path: docs/dev/connectors/kafka.md ## @@ -134,7 +134,7 @@ Then instantiate the new source (`FlinkKafkaConsumer`) and sink (`FlinkKafkaProd ## Kafka Consumer -Flink's Kafka consumer is called `FlinkKafkaConsumer08` (or `09` for Kafka 0.9.0.x versions, etc.). It provides access to one or more Kafka topics. +Flink's Kafka consumer is called `FlinkKafkaConsumer08` (or `09` for Kafka 0.9.0.x versions, etc. Note that since modern Kafka connector, it will not carry the version number of Kafka.). It provides access to one or more Kafka topics. Review comment: (or `09` for Kafka 0.9.0.x versions, etc. or just `FlinkKafkaConsumer` for Kafka >= 1.0.0 versions). ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663898#comment-16663898 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228219659 ## File path: docs/dev/connectors/kafka.md ## @@ -650,13 +650,13 @@ into a Kafka topic. for more explanation. - Kafka 0.11 + Kafka 0.11 and newer -With Flink's checkpointing enabled, the `FlinkKafkaProducer011` can provide +With Flink's checkpointing enabled, the `FlinkKafkaProducer011` (for modern Kafka connector, it is `FlinkKafkaProducer`.) can provide exactly-once delivery guarantees. Besides enabling Flink's checkpointing, you can also choose three different modes of operating -chosen by passing appropriate `semantic` parameter to the `FlinkKafkaProducer011`: +chosen by passing appropriate `semantic` parameter to the `FlinkKafkaProducer011` (for modern Kafka connector, it is `FlinkKafkaProducer`.): Review comment: ditto? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663868#comment-16663868 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-433089952 @pnowojski thanks for your suggestion, updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[
https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663686#comment-16663686
]
ASF GitHub Bot commented on FLINK-10599:
pnowojski commented on a change in pull request #6889:
[FLINK-10599][Documentation] Provide documentation for the modern kafka
connector
URL: https://github.com/apache/flink/pull/6889#discussion_r228153790
##
File path: docs/dev/connectors/kafka.md
##
@@ -100,6 +108,30 @@ Note that the streaming connectors are currently not part
of the binary distribu
* Follow the instructions from [Kafka's
quickstart](https://kafka.apache.org/documentation.html#quickstart) to download
the code and launch a server (launching a Zookeeper and a Kafka server is
required every time before starting the application).
* If the Kafka and Zookeeper servers are running on a remote machine, then the
`advertised.host.name` setting in the `config/server.properties` file must be
set to the machine's IP address.
+## Modern Kafka Connector
+
+Starting with Flink 1.7, there is a new Kafka connector that does not track a
specific Kafka major version. Rather, it tracks the latest version of Kafka at
the time of the Flink release.
+
+If your Kafka broker version is 1.0.0 or newer, you should use this Kafka
connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you
should use the connector corresponding to the broker version.
+
+### Compatibility
+
+The modern Kafka connector is compatible with older and newer Kafka brokers
through the compatibility guarantees of the Kafka client API and broker. The
modern Kafka client is compatible with broker versions 0.10.0 or later,
depending on the features used. For details on Kafka compatibility, please
refer to the [Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).
+
+### Usage
+
+The use of the modern Kafka connector add a dependency to it:
+
+{% highlight xml %}
+
+ org.apache.flink
+ flink-connector-kafka{{ site.scala_version_suffix }}
+ {{ site.version }}
+
+{% endhighlight %}
+
+Then instantiate the new source (`FlinkKafkaConsumer`) and sink
(`FlinkKafkaProducer`). The API is the backwards compatible with the older
Kafka connectors.
+
## Kafka Consumer
Flink's Kafka consumer is called `FlinkKafkaConsumer08` (or `09` for Kafka
0.9.0.x versions, etc.). It provides access to one or more Kafka topics.
Review comment:
Should this line also be adjusted? I think we should also mention here that
"modern" connector brakes the naming convention.
Same applies to similar sentence later for the producer:
```
Flink’s Kafka Producer is called FlinkKafkaProducer011 (or 010 for Kafka
0.10.0.x versions, etc.). It allows writing a stream of records to one or more
Kafka topics.
```
Also in `Kafka Producers and Fault Tolerance` section, rename the title
`Kafka 0.11` to `Kafka 0.11 and newer`, and also adopt the content that section
(the name `FlinkKafkaProducer011 ` is used there frequently and with modern
connector it should also be adjusted).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide documentation for the modern kafka connector
>
>
> Key: FLINK-10599
> URL: https://issues.apache.org/jira/browse/FLINK-10599
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663687#comment-16663687 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228152682 ## File path: docs/dev/connectors/kafka.md ## @@ -80,6 +80,14 @@ For most users, the `FlinkKafkaConsumer08` (part of `flink-connector-kafka`) is 0.11.x Since 0.11.x Kafka does not support scala 2.10. This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging";>Kafka transactional messaging to provide exactly once semantic for the producer. + +flink-connector-kafka_2.11 +1.7.0 +FlinkKafkaConsumer +FlinkKafkaProducer +>= 1.x Review comment: `>= 1.0.0` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663684#comment-16663684 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228150529 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector Review comment: Both of those names can be confusing for the user. Maybe we should name this section as `Kafka 0.11+ Connector`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663685#comment-16663685 ] ASF GitHub Bot commented on FLINK-10599: pnowojski commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r228151680 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,30 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. + +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. + +### Compatibility + +The modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka client is compatible with broker versions 0.10.0 or later, depending on the features used. For details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). Review comment: Have we tested that this new connector works with Kafka 0.10? Definitely not in `exacty_once` mode. Maybe it does work with `at_least_once` mode, but if we haven't tested that, I would refrain from saying it. So I would rephrase this to: ``` The modern Kafka connector is compatible with broker versions 0.11.0 or later ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662164#comment-16662164 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-432622691 @zentol @aljoscha @pnowojski Can you give a final check about this documentation? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660476#comment-16660476 ] ASF GitHub Bot commented on FLINK-10599: zentol commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227351375 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,30 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. + +If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. + +### Compatibility + +The the modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka client is compatible with broker versions 0.10.0 or later, depending on the features used. Fo details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). Review comment: double "the"; typo: Fo -> For This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660485#comment-16660485 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-432214452 @zentol updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[
https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660477#comment-16660477
]
ASF GitHub Bot commented on FLINK-10599:
zentol commented on a change in pull request #6889:
[FLINK-10599][Documentation] Provide documentation for the modern kafka
connector
URL: https://github.com/apache/flink/pull/6889#discussion_r227351575
##
File path: docs/dev/connectors/kafka.md
##
@@ -100,6 +108,30 @@ Note that the streaming connectors are currently not part
of the binary distribu
* Follow the instructions from [Kafka's
quickstart](https://kafka.apache.org/documentation.html#quickstart) to download
the code and launch a server (launching a Zookeeper and a Kafka server is
required every time before starting the application).
* If the Kafka and Zookeeper servers are running on a remote machine, then the
`advertised.host.name` setting in the `config/server.properties` file must be
set to the machine's IP address.
+## Modern Kafka Connector
+
+Starting with Flink 1.7, there is a new Kafka connector that does not track a
specific Kafka major version. Rather, it tracks the latest version of Kafka at
the time of the Flink release.
+
+If your Kafka broker version is 1.0.0 or newer, you should use this Kafka
connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you
should use the connector corresponding to the broker version.
+
+### Compatibility
+
+The the modern Kafka connector is compatible with older and newer Kafka
brokers through the compatibility guarantees of the Kafka client API and
broker. The modern Kafka client is compatible with broker versions 0.10.0 or
later, depending on the features used. Fo details on Kafka compatibility,
please refer to the [Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).
+
+### Usage
+
+The use of the modern Kafka connector add a dependency to it:
+
+{% highlight xml %}
+
+ org.apache.flink
+ flink-connector-kafka{{ site.scala_version_suffix }}
+ {{site.version }}
Review comment:
sync number of spaces around `site.version`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide documentation for the modern kafka connector
>
>
> Key: FLINK-10599
> URL: https://issues.apache.org/jira/browse/FLINK-10599
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660451#comment-16660451 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-432206118 Hi @eliaslevy Thank you for reviewing it, I have adopted most of your suggestions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660449#comment-16660449 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227342714 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector Review comment: I don't object, what do you think? cc @aljoscha This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659547#comment-16659547 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227031828 ## File path: docs/dev/connectors/kafka.md ## @@ -80,6 +80,14 @@ For most users, the `FlinkKafkaConsumer08` (part of `flink-connector-kafka`) is 0.11.x Since 0.11.x Kafka does not support scala 2.10. This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging";>Kafka transactional messaging to provide exactly once semantic for the producer. + +flink-connector-kafka_2.11 +1.7.0 +FlinkKafkaConsumer +FlinkKafkaProducer +1.x,2.x +This is a modern kafka connector that does not bind to a fixed version of the Kafka client, it only tracks the latest version of the Kafka client. Review comment: You may want to rephrase this. change between releases to track the latest Kafka version. ```suggestion This Kafka connector attempts to track the latest version of the Kafka client. The version of the client it uses may change between Flink releases. Modern Kafka clients are backwards compatible with broker versions 0.10.0 or later. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[
https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659542#comment-16659542
]
ASF GitHub Bot commented on FLINK-10599:
eliaslevy commented on a change in pull request #6889:
[FLINK-10599][Documentation] Provide documentation for the modern kafka
connector
URL: https://github.com/apache/flink/pull/6889#discussion_r227039972
##
File path: docs/dev/connectors/kafka.md
##
@@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part
of the binary distribu
* Follow the instructions from [Kafka's
quickstart](https://kafka.apache.org/documentation.html#quickstart) to download
the code and launch a server (launching a Zookeeper and a Kafka server is
required every time before starting the application).
* If the Kafka and Zookeeper servers are running on a remote machine, then the
`advertised.host.name` setting in the `config/server.properties` file must be
set to the machine's IP address.
+## Modern Kafka Connector
+
+Starting with Flink 1.7, it will provide a new connector for the modern Kafka.
It is different from any previous Kafka connector, it does not bind a fixed
version of Kafka client, but it will track and support the latest version of
Kafka Client.
+The modern Kafka connector does not rely on any older version of the kafka
connector and relies only on the underlying implementation of the kafka
connector, which allows it to get rid of the historical burden and keep up with
the latest version of the Kafka client.
+
+Note that if your version of Kafka Server exceeds 0.11, then we recommend you
use the modern Kafka connector. The specific version of the kafka connector
will continue to maintain and fix related issues. When you use the old version
of Kafka (for example, 0.11, 0.10, 0.9, 0.8), we still recommend that you use
the connector corresponding to the corresponding version, because they are
precisely matched.
+
+### Compatibility
+
+The compatibility of the modern Kafka connector is completely dependent on the
backward compatibility guarantee of the Kafka Client API (Kafka supports the
latest Client API to access the old version of Kafka Server. For more details
on Kafka compatibility, please refer to the [official Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility)).
Therefore, as long as the connector is upgraded and supported to the latest
Kafka Client version, it can provide the same compatibility guarantee as the
Kafka Client.
+
+### Usage
+
+The use of the modern Kafka connector is very simple. We only need to
introduce dependencies on it:
+
+{% highlight xml %}
+
+ org.apache.flink
+ flink-connector-kafka{{ site.scala_version_suffix }}
+ {{site.version }}
+
+{% endhighlight %}
+
+Then instantiate the new source (`FlinkKafkaConsumer`) and sink
(`FlinkKafkaProducer`). As for the API, it is fully compatible with other Kafka
connectors that already exist. So in theory, you don't need to make any changes
to it to make your program work.
Review comment:
```suggestion
Then instantiate the new source (`FlinkKafkaConsumer`) and sink
(`FlinkKafkaProducer`). The API is the backwards compatible with the older
Kafka connectors.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide documentation for the modern kafka connector
>
>
> Key: FLINK-10599
> URL: https://issues.apache.org/jira/browse/FLINK-10599
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659544#comment-16659544 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227033560 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, it will provide a new connector for the modern Kafka. It is different from any previous Kafka connector, it does not bind a fixed version of Kafka client, but it will track and support the latest version of Kafka Client. +The modern Kafka connector does not rely on any older version of the kafka connector and relies only on the underlying implementation of the kafka connector, which allows it to get rid of the historical burden and keep up with the latest version of the Kafka client. Review comment: I'd probably remove this paragraph. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659539#comment-16659539 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227040788 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector Review comment: I am wondering if instead of "Modern" we should just call it "unversioned". This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659541#comment-16659541 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227039043 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, it will provide a new connector for the modern Kafka. It is different from any previous Kafka connector, it does not bind a fixed version of Kafka client, but it will track and support the latest version of Kafka Client. +The modern Kafka connector does not rely on any older version of the kafka connector and relies only on the underlying implementation of the kafka connector, which allows it to get rid of the historical burden and keep up with the latest version of the Kafka client. + +Note that if your version of Kafka Server exceeds 0.11, then we recommend you use the modern Kafka connector. The specific version of the kafka connector will continue to maintain and fix related issues. When you use the old version of Kafka (for example, 0.11, 0.10, 0.9, 0.8), we still recommend that you use the connector corresponding to the corresponding version, because they are precisely matched. + +### Compatibility + +The compatibility of the modern Kafka connector is completely dependent on the backward compatibility guarantee of the Kafka Client API (Kafka supports the latest Client API to access the old version of Kafka Server. For more details on Kafka compatibility, please refer to the [official Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility)). Therefore, as long as the connector is upgraded and supported to the latest Kafka Client version, it can provide the same compatibility guarantee as the Kafka Client. + +### Usage + +The use of the modern Kafka connector is very simple. We only need to introduce dependencies on it: Review comment: ```suggestion The use of the modern Kafka connector add a dependency to it: ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659540#comment-16659540 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227040267 ## File path: docs/dev/connectors/kafka.md ## @@ -80,6 +80,14 @@ For most users, the `FlinkKafkaConsumer08` (part of `flink-connector-kafka`) is 0.11.x Since 0.11.x Kafka does not support scala 2.10. This connector supports https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging";>Kafka transactional messaging to provide exactly once semantic for the producer. + +flink-connector-kafka_2.11 +1.7.0 +FlinkKafkaConsumer +FlinkKafkaProducer +1.x,2.x Review comment: ```suggestion >= 1.x ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659546#comment-16659546 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227038721 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, it will provide a new connector for the modern Kafka. It is different from any previous Kafka connector, it does not bind a fixed version of Kafka client, but it will track and support the latest version of Kafka Client. +The modern Kafka connector does not rely on any older version of the kafka connector and relies only on the underlying implementation of the kafka connector, which allows it to get rid of the historical burden and keep up with the latest version of the Kafka client. + +Note that if your version of Kafka Server exceeds 0.11, then we recommend you use the modern Kafka connector. The specific version of the kafka connector will continue to maintain and fix related issues. When you use the old version of Kafka (for example, 0.11, 0.10, 0.9, 0.8), we still recommend that you use the connector corresponding to the corresponding version, because they are precisely matched. + +### Compatibility + +The compatibility of the modern Kafka connector is completely dependent on the backward compatibility guarantee of the Kafka Client API (Kafka supports the latest Client API to access the old version of Kafka Server. For more details on Kafka compatibility, please refer to the [official Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility)). Therefore, as long as the connector is upgraded and supported to the latest Kafka Client version, it can provide the same compatibility guarantee as the Kafka Client. Review comment: ```suggestion The the modern Kafka connector is compatible with older and newer Kafka brokers through the compatibility guarantees of the Kafka client API and broker. The modern Kafka client is compatible with broker versions 0.10.0 or later, depending on the features used. Fo details on Kafka compatibility, please refer to the [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility). ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659543#comment-16659543 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227035809 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, it will provide a new connector for the modern Kafka. It is different from any previous Kafka connector, it does not bind a fixed version of Kafka client, but it will track and support the latest version of Kafka Client. +The modern Kafka connector does not rely on any older version of the kafka connector and relies only on the underlying implementation of the kafka connector, which allows it to get rid of the historical burden and keep up with the latest version of the Kafka client. + +Note that if your version of Kafka Server exceeds 0.11, then we recommend you use the modern Kafka connector. The specific version of the kafka connector will continue to maintain and fix related issues. When you use the old version of Kafka (for example, 0.11, 0.10, 0.9, 0.8), we still recommend that you use the connector corresponding to the corresponding version, because they are precisely matched. Review comment: ```suggestion If your Kafka broker version is 1.0.0 or newer, you should use this Kafka connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), you should use the connector corresponding to the broker version. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659545#comment-16659545 ] ASF GitHub Bot commented on FLINK-10599: eliaslevy commented on a change in pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#discussion_r227033074 ## File path: docs/dev/connectors/kafka.md ## @@ -100,6 +108,31 @@ Note that the streaming connectors are currently not part of the binary distribu * Follow the instructions from [Kafka's quickstart](https://kafka.apache.org/documentation.html#quickstart) to download the code and launch a server (launching a Zookeeper and a Kafka server is required every time before starting the application). * If the Kafka and Zookeeper servers are running on a remote machine, then the `advertised.host.name` setting in the `config/server.properties` file must be set to the machine's IP address. +## Modern Kafka Connector + +Starting with Flink 1.7, it will provide a new connector for the modern Kafka. It is different from any previous Kafka connector, it does not bind a fixed version of Kafka client, but it will track and support the latest version of Kafka Client. Review comment: ```suggestion Starting with Flink 1.7, there is a new Kafka connector that does not track a specific Kafka major version. Rather, it tracks the latest version of Kafka at the time of the Flink release. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658171#comment-16658171 ] ASF GitHub Bot commented on FLINK-10599: yanghua commented on issue #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889#issuecomment-431659542 Hi @aljoscha I provided a simple introduction and documentation for the modern kafka connector. Considering that I am not a native English user, you are welcome to point out where the documentation is inappropriate. Also, if it is missing something, pelase feel free to tell me. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658169#comment-16658169 ] ASF GitHub Bot commented on FLINK-10599: yanghua opened a new pull request #6889: [FLINK-10599][Documentation] Provide documentation for the modern kafka connector URL: https://github.com/apache/flink/pull/6889 ## What is the purpose of the change *This pull request provides documentation for the modern kafka connector* ## Brief change log - *Provide documentation for the modern kafka connector* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655153#comment-16655153 ] Aljoscha Krettek commented on FLINK-10599: -- We can always merge documentation, because the documentation website doesn't depend on a specific release but is always the documentation from the branch. > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10599) Provide documentation for the modern kafka connector
[ https://issues.apache.org/jira/browse/FLINK-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654949#comment-16654949 ] vinoyang commented on FLINK-10599: -- [~aljoscha] and [~till.rohrmann] If I start this issue now, is it possible to be merged into 1.7? If the probability is small, then I will deal with other higher priority issues. Or what should be provided about this document, do you have any suggestions? > Provide documentation for the modern kafka connector > > > Key: FLINK-10599 > URL: https://issues.apache.org/jira/browse/FLINK-10599 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
