[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678068#comment-16678068 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436592966 AFAIK, many test cases are unstable. Recently, I have encountered Travis failures caused by various problems in multiple modules (including library, connector, e2e test). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677958#comment-16677958 ] ASF GitHub Bot commented on FLINK-10600: shaoxuan-wang commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436572212 > @yanghua if the tests are not stable for 2.0 connector, they should be fixed. > > @shaoxuan-wang what test failure do you have in mind? @pnowojski . I cannot remember for now. But I a pretty sure it is either "Modern Kafka end-to-end test" or "Kafka end-to-end test". I quickly checked gitlog and find this pr on top of it. When I rerun the test it passed. I guess one of these test is not stable. I will report it next when I see the failure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677929#comment-16677929 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436566067 @yanghua if the tests are not stable for 2.0 connector, they should be fixed. @shaoxuan-wang what test failure do you have in mind? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676692#comment-16676692 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436235491 @shaoxuan-wang Can you share your error log with us? The test cases related to kafka connector are not very stable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676690#comment-16676690 ] ASF GitHub Bot commented on FLINK-10600: shaoxuan-wang commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436234220 @pnowojski @yanghuaI got this test failure when I rebase the flink master to my own repository which is connected with travis-ci.org. And I did not find the origin travis test result in this thread. Is it possible this is a flaky test? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676430#comment-16676430
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski closed pull request #6924: [FLINK-10600] Provide End-to-end test
cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-connectors/flink-connector-kafka/pom.xml
b/flink-connectors/flink-connector-kafka/pom.xml
index cfc92c00131..617f1232e35 100644
--- a/flink-connectors/flink-connector-kafka/pom.xml
+++ b/flink-connectors/flink-connector-kafka/pom.xml
@@ -52,6 +52,10 @@ under the License.
org.apache.kafka
kafka_${scala.binary.version}
+
+ org.apache.kafka
+ kafka-clients
+
diff --git a/flink-dist/src/main/assemblies/bin.xml
b/flink-dist/src/main/assemblies/bin.xml
index 89228b39271..e1adab537d9 100644
--- a/flink-dist/src/main/assemblies/bin.xml
+++ b/flink-dist/src/main/assemblies/bin.xml
@@ -187,6 +187,34 @@ under the License.
+
+
+
../flink-examples/flink-examples-streaming-kafka/target
+ examples/streaming
+ 0644
+
+ *.jar
+
+
+
flink-examples-streaming-kafka*.jar
+ original-*.jar
+
+
+
+
+
+
../flink-examples/flink-examples-streaming-kafka-0.10/target
+ examples/streaming
+ 0644
+
+ *.jar
+
+
+
flink-examples-streaming-kafka*.jar
+ original-*.jar
+
+
+
../flink-libraries/flink-gelly-examples/target
diff --git a/flink-end-to-end-tests/run-pre-commit-tests.sh
b/flink-end-to-end-tests/run-pre-commit-tests.sh
index 15b9b152f77..3e3494a491a 100755
--- a/flink-end-to-end-tests/run-pre-commit-tests.sh
+++ b/flink-end-to-end-tests/run-pre-commit-tests.sh
@@ -54,6 +54,7 @@ run_test "Batch Python Wordcount end-to-end test"
"$END_TO_END_DIR/test-scripts/
run_test "Streaming Python Wordcount end-to-end test"
"$END_TO_END_DIR/test-scripts/test_streaming_python_wordcount.sh"
run_test "Wordcount end-to-end test"
"$END_TO_END_DIR/test-scripts/test_batch_wordcount.sh"
run_test "Kafka end-to-end test"
"$END_TO_END_DIR/test-scripts/test_streaming_kafka010.sh"
+run_test "Modern Kafka end-to-end test"
"$END_TO_END_DIR/test-scripts/test_streaming_kafka.sh"
run_test "class loading end-to-end test"
"$END_TO_END_DIR/test-scripts/test_streaming_classloader.sh"
run_test "Shaded Hadoop S3A end-to-end test"
"$END_TO_END_DIR/test-scripts/test_shaded_hadoop_s3a.sh"
run_test "Shaded Presto S3 end-to-end test"
"$END_TO_END_DIR/test-scripts/test_shaded_presto_s3.sh"
diff --git a/flink-end-to-end-tests/test-scripts/kafka-common.sh
b/flink-end-to-end-tests/test-scripts/kafka-common.sh
index 2dc58f7305a..dedfe5208b1 100644
--- a/flink-end-to-end-tests/test-scripts/kafka-common.sh
+++ b/flink-end-to-end-tests/test-scripts/kafka-common.sh
@@ -17,6 +17,8 @@
# limitations under the License.
+set -e
+set -u
set -o pipefail
if [[ -z $TEST_DATA_DIR ]]; then
@@ -24,15 +26,19 @@ if [[ -z $TEST_DATA_DIR ]]; then
exit 1
fi
-KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-0.10.2.0
-CONFLUENT_DIR=$TEST_DATA_DIR/confluent-3.2.0
+KAFKA_VERSION="$1"
+CONFLUENT_VERSION="$2"
+CONFLUENT_MAJOR_VERSION="$3"
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-$KAFKA_VERSION
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-$CONFLUENT_VERSION
SCHEMA_REGISTRY_PORT=8082
SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
function setup_kafka_dist {
# download Kafka
mkdir -p $TEST_DATA_DIR
-
KAFKA_URL="https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz;
+
KAFKA_URL="https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz;
echo "Downloading Kafka from $KAFKA_URL"
curl
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675278#comment-16675278 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435904341 updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675274#comment-16675274 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435902208 @pnowojski Oh... yes, I take it for granted that this is streaming's example and does not rely on batch API. It's my fault. I'll fix it right away. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675268#comment-16675268 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435900933 I meant `flink-java` because you are explicitly using `ParameterTool` class from that package. Aren't you? You are not using anything from `flink-streaming-scala`, so you shouldn't depend on it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675259#comment-16675259 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435898739 @pnowojski updated (squashed commits for `pull --rebase`) and I think what do mean is `flink-streaming-java` and `flink-streaming-scala`, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675260#comment-16675260 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435898739 @pnowojski updated (squashed commits for `pull --rebase`) and I think what you mean is `flink-streaming-java` and `flink-streaming-scala`, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675219#comment-16675219 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r230764838 ## File path: flink-examples/flink-examples-streaming-kafka-0.10/pom.xml ## @@ -0,0 +1,76 @@ + + +http://maven.apache.org/POM/4.0.0; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + + flink-examples + org.apache.flink + 1.7-SNAPSHOT + + 4.0.0 + + flink-examples-streaming-kafka-0.10 + + + Review comment: It will be better if you do not relay on transitive dependencies here and explicitly list `flink-streaming-java` and `flink-java` dependencies in the pom. Now it works because those modules are transitive dependency of `flink-connector-kafka-0.10` while `Kafka010Example` explicitly uses/depends on them. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675218#comment-16675218 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r230765090 ## File path: flink-examples/flink-examples-streaming-kafka/pom.xml ## @@ -0,0 +1,78 @@ + + +http://maven.apache.org/POM/4.0.0; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + + flink-examples + org.apache.flink + 1.7-SNAPSHOT + + 4.0.0 + + flink-examples-streaming-kafka + flink-examples-streaming-kafka + + + Review comment: ditto: add `flink-streaming-java` and `flink-java`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675209#comment-16675209
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230763738
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
I think the new flat organisation that you proposed is fine, makes it more
fine grained and easier to understand/re-use by the user.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675211#comment-16675211
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230763738
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
I think the new flat organisation that you proposed is fine, makes it more
fine grained and easier to understand/re-use by the user. So good change :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674845#comment-16674845
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230673105
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
@pnowojski and @aljoscha It is hoped that the end-to-end testing for the
modern Kafka connector can be merged before the 1.7.0 release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672680#comment-16672680
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230287880
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
Let us work closely together today to complete this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672677#comment-16672677
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230287447
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
@pnowojski I tried the solution you said, but it needs to convert the
existing `flink-examples-streaming` into a parent module (container module).
There are two problems with this:
* The previous packaging mode of `flink-examples-streaming` is jar. After
modification, its packing mode is pom, but it will pull the pom file in remote
maven every time before packaging, which causes local packaging to fail.
* This approach will introduce a multi-layered container module, which
breaks the current model of the flink project.
I prefer a flat organizational model, that is, only one layer of the
container module.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671547#comment-16671547
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230025953
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
OK, agree.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671544#comment-16671544
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230025234
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
But it also has other types of examples, such as `join`, `ml`, `window`...
If it contains submodules, how do you handle these examples?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671541#comment-16671541
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230024940
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
In that case you should probably fix it in `flink-kafka-connector/pom.xml`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671536#comment-16671536
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230024231
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
No, sorry about a typo. I was thinking about creating a submodule in the
existing one `flink-examples/flink-examples-streaming/`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671525#comment-16671525
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230022088
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
@pnowojski I think currently there is not a module named
`flink-streaming-examples` but a module named `flink-examples-streaming` which
is not a parent module (and contains many kind of examples). Do you mean that I
need to create a module called `flink-streaming-examples`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671471#comment-16671471
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r230007754
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
```
03:59:30.102 [WARNING]
Dependency convergence error for org.apache.kafka:kafka-clients:0.10.2.1
paths to dependency are:
+-org.apache.flink:flink-kafka-test:1.7-SNAPSHOT
+-org.apache.flink:flink-connector-kafka_2.11:1.7-SNAPSHOT
+-org.apache.flink:flink-connector-kafka-base_2.11:1.7-SNAPSHOT
+-org.apache.kafka:kafka-clients:0.10.2.1
and
+-org.apache.flink:flink-kafka-test:1.7-SNAPSHOT
+-org.apache.flink:flink-connector-kafka_2.11:1.7-SNAPSHOT
+-org.apache.kafka:kafka-clients:2.0.0
03:59:30.103 [WARNING] Rule 0:
org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
Failed while enforcing releasability. See above detailed error message.
```
details : https://api.travis-ci.org/v3/job/449173244/log.txt
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671395#comment-16671395
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229993379
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
What was the error? We should definitely look into it, since we shouldn't
expect users to use the same hack in their poms.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671359#comment-16671359
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229985611
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
OK, it sounds a good solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671356#comment-16671356
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229985144
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
I didn't introduce it at first, this was done to explicitly exclude the base
module's dependency on the lower version of the kafka client. Otherwise it will
cause a compilation error.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671343#comment-16671343
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229975091
##
File path: flink-end-to-end-tests/flink-kafka-test/pom.xml
##
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.7-SNAPSHOT
+
+ 4.0.0
+
+ flink-kafka-test
+ flink-kafka-test
+
+
+
+ org.apache.flink
+ flink-kafka-test-base
+ ${project.version}
+
+
+
+ org.apache.flink
+
flink-connector-kafka-base_${scala.binary.version}
Review comment:
I don't think that you need this dependency here at all. Definitely it
shouldn't be here, since we do not expect our users to include it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671344#comment-16671344
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229983121
##
File path:
flink-end-to-end-tests/flink-kafka-test-base/src/main/java/org/apache/flink/kafka/tests/KafkaEvent.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kafka.tests;
+
+/**
+ * The event type used in the {@link Kafka010Example}.
+ *
+ * This is a Java POJO, which Flink recognizes and will allow "by-name"
field referencing
+ * when keying a {@link org.apache.flink.streaming.api.datastream.DataStream}
of such a type.
+ * For a demonstration of this, see the code in {@link Kafka010Example}.
+ */
+public class KafkaEvent {
Review comment:
I have talked with @tzulitai and we concluded that to avoid this duplication
it would be best to:
- Move submodule (and rename) `flink-kafka-test-base` to
`flink-streaming-examples/flink-kafka-example-base`
- Kafka version specifics (`flink-streaming-examples/flink-kafka-example`
and `flink-streaming-examples/flink-kafka-0.10-example`) extend that base
- End to end tests just use the example jobs for testing
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671163#comment-16671163
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229950857
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
@pnowojski I have split the modern kafka connector into a separate module,
but I have not restored the code for the flink-examples-streaming module.
Please review again (test failure is irrelevant).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670361#comment-16670361
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229773839
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
@pnowojski @tzulitai The result is not as expected (it failed), and even if
we use `artifactSet / exculde` to exclude other connectors, the dependency on
kafka-client is still wrong. So I'll take your advice and split the test
implementation of the modern Kafka connector into separate modules. But because
of the logical similarity, do you recommend that I introduce a base
implementation (in a new base module)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670015#comment-16670015
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229669907
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
@pnowojski I ruled it out because I think the 2.0 version of the connector
relies on the kafka client. But then I chose to exclude different connectors
for different packaging modes. I can try it out and undo this code to see if it
will cause problems.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669988#comment-16669988
]
ASF GitHub Bot commented on FLINK-10600:
tzulitai commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229659492
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
I also think that the Kafka 2.0 example should be in a separate module.
Take for example our Elasticsearch E2E tests: each version of the
Elasticsearch job lives within its own module, so that we don't need to do any
extra things with the dependency packaging.
On the other hand, as you can see, the Elasticsearch jobs live under the
`flink-end-to-end-tests` module.
Perhaps it would make sense to do the same for Kafka, instead of relying on
the released example jobs as we do right now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669846#comment-16669846
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide some error
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669847#comment-16669847
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide or cause some errors
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669842#comment-16669842
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622248
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
Add comment `# Terminate Kafka process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669841#comment-16669841
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622276
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
- PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
`# Terminate QuorumPeerMain process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669843#comment-16669843
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here? end-to-end tests should be the
final validation that our code works and having to modify dependencies doesn't
sound good and might hide some error.
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669808#comment-16669808 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434615379 @pnowojski Thank you for your guidance, now the test can pass. I submitted the last change, can you help me to review it again? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669774#comment-16669774
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski edited a comment on issue #6924: [FLINK-10600] Provide End-to-end
test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
if it works, then add comment above
```
# Terminate QuorumPeerMain process if it still exists
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669772#comment-16669772
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases
for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669517#comment-16669517
]
ASF GitHub Bot commented on FLINK-10600:
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test
cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434539461
@pnowojski I can give you more details :
added `set -x` and `set -e`:
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ grep -i 'kafka\.Kafka'
++ jps -vl
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=55405
+ '[' '!' -z 55405 ']'
+ kill -s TERM 55405
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
[FAIL] 'Modern Kafka end-to-end test' failed after 0 minutes and 57 seconds!
Test exited with exit code 1
Stopping taskexecutor daemon (pid: 56949) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 56518) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
```
But, just add `set -x`
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ jps -vl
++ grep -i 'kafka\.Kafka'
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=62800
+ '[' '!' -z 62800 ']'
+ kill -s TERM 62800
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
+ '[' '!' -z '' ']'
+ mv -f
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml.bak
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml
[PASS] 'Modern Kafka end-to-end test' passed after 1 minutes and 0 seconds!
Test exited with exit code 0.
Stopping taskexecutor daemon (pid: 64351) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 63917) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
Deleted all files under
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/log/
Deleted
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N
[PASS] All tests passed
```
the problem (see comment in this function):
```shell
function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
# here
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669516#comment-16669516
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases
for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434539461
@pnowojski I can give you more details :
added `set -x` and `set -e`:
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ grep -i 'kafka\.Kafka'
++ jps -vl
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=55405
+ '[' '!' -z 55405 ']'
+ kill -s TERM 55405
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
[FAIL] 'Modern Kafka end-to-end test' failed after 0 minutes and 57 seconds!
Test exited with exit code 1
Stopping taskexecutor daemon (pid: 56949) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 56518) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
```
But, just add `set -x`
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ jps -vl
++ grep -i 'kafka\.Kafka'
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=62800
+ '[' '!' -z 62800 ']'
+ kill -s TERM 62800
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
+ '[' '!' -z '' ']'
+ mv -f
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml.bak
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml
[PASS] 'Modern Kafka end-to-end test' passed after 1 minutes and 0 seconds!
Test exited with exit code 0.
Stopping taskexecutor daemon (pid: 64351) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 63917) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
Deleted all files under
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/log/
Deleted
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N
[PASS] All tests passed
```
the problem :
```shell
function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
# here
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668992#comment-16668992 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434372198 @pnowojski I just tested again, added `set -x` on the top of `kafka-common.sh`, `test_streaming_kafka_common.sh` and `test_streaming_kafka.sh`. Everything is OK and it shown `[PASS]` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668936#comment-16668936 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434361242 > shell function `get_partition_end_offset` in kafka-common.sh : > > ``` > $KAFKA_DIR/bin/kafka-consumer-groups.sh --describe --group dummy-consumer --bootstrap-server localhost:9092 > ``` > always and just print : > > ``` > Consumer group 'dummy-consumer' has no active members. > ``` @pnowojski This is the reason for the shell error. Now I changed the way of getting the offset, so it fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668926#comment-16668926 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434358753 @pnowojski I have debugged locally, and the new changes have fixed the shell error. Before fixing, even if I do not add `set -e`, it will report an error. Now, I add `set -e`, which still doesn't report any errors, but the result doesn't show 'pass'. When I remove the `set -e`, the result will show that the test is passed. Considering that there was no `set-e' in these scripts before, I think it was this sentence that caused the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668929#comment-16668929 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434358753 @pnowojski I have debugged locally, and the new changes have fixed the shell error. Before fixing, even if I do not add `set -e`, it will report an error. Now, I add `set -e`, which still doesn't report any errors, but the result doesn't show 'pass'. When I remove the `set -e`, the result will show that the test is passed. Considering that there was no `set -e` in these scripts before, I think it was this sentence that caused the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668924#comment-16668924 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434358753 @pnowojski I have debugged locally, and the new changes have fixed the shell error. Before fixing, even if I do not add `set -e`, it will report an error. Now, I add `set-e', which still doesn't report any errors, but the result doesn't show 'pass'. When I remove the `set -e`, the result will show that the test is passed. Considering that there was no `set-e' in these scripts before, I think it was this sentence that caused the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668906#comment-16668906 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434354247 @yanghua it's not OK if we do not add `set -e`. The error would be still there, but it would just be silently ignored. You might want to to `set -x` as well for debugging, to print the commands that are being executed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668884#comment-16668884 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434348494 @pnowojski You can check the [old Travis log](https://api.travis-ci.org/v3/job/448260257/log.txt). Just search : ``` [FAIL] 'Kafka end-to-end test' failed after 0 minutes and 57 seconds! Test exited with exit code 1 ``` The shell script does not report the error, but the result would show this information. If we do not add `set -e`, it would be OK. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668746#comment-16668746 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434311640 How do they fail? I would say they should be fixed and if errors are expected, they should be handled. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668527#comment-16668527 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434255369 @pnowojski We cannot add `set -e` to top of the modified files. This action would cause the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668471#comment-16668471
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229245803
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -120,14 +126,7 @@ function get_partition_end_offset {
local topic=$1
local partition=$2
- # first, use the console consumer to produce a dummy consumer group
- read_messages_from_kafka 0 $topic dummy-consumer
-
- # then use the consumer offset utility to get the LOG_END_OFFSET value for
the specified partition
- $KAFKA_DIR/bin/kafka-consumer-groups.sh --describe --group dummy-consumer
--bootstrap-server localhost:9092 2> /dev/null \
-| grep "$topic \+$partition" \
-| tr -s " " \
-| cut -d " " -f 4
+ $KAFKA_DIR/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic $topic --partitions $partition --time -1 | cut -d ":" -f
3
Review comment:
I changed the way of getting the offset. @pnowojski
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668459#comment-16668459 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434241140 I would guess "group" from here: ``` kafka-common.sh: $KAFKA_DIR/bin/kafka-consumer-groups.sh --describe --group dummy-consumer --bootstrap-server localhost:9092 2> /dev/null \ ``` is out of sync with this: ``` test_streaming_kafka010.sh: --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id myconsumer --auto.offset.reset earliest \ ``` Just make `group` in `kafka-common.sh` configurable via a parameter, create it as a constant in `test_streaming_kafka.sh` and pass this constant to `get_partition_end_offset`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668243#comment-16668243 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434190482 @tzulitai shell function `get_partition_end_offset` in kafka-common.sh : ``` $KAFKA_DIR/bin/kafka-consumer-groups.sh --describe --group dummy-consumer --bootstrap-server localhost:9092 ``` always and just print : ``` Consumer group 'dummy-consumer' has no active members. ``` Any suggestion? cc @pnowojski This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668028#comment-16668028
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229159258
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -538,6 +550,11 @@ under the License.
shade
+
+
+
org.apache.flink:flink-connector-kafka_${scala.binary.version}
Review comment:
yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667290#comment-16667290
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228949560
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -538,6 +550,11 @@ under the License.
shade
+
+
+
org.apache.flink:flink-connector-kafka_${scala.binary.version}
Review comment:
Ok, right. We can only exclude the modern one and `0.11`? And we can still
exclude everything from "modern" `KafkaExample`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667191#comment-16667191
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228923208
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -538,6 +550,11 @@ under the License.
shade
+
+
+
org.apache.flink:flink-connector-kafka_${scala.binary.version}
Review comment:
We can't exclude these connectors, which is reported in my travis because
there is a cascading dependency between the previous connectors. If you exclude
it, you will get an error:
```
java.lang.NoClassDefFoundError:
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.flink.streaming.examples.kafka.Kafka010Example.main(Kafka010Example.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 29 more
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667188#comment-16667188 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433904230 Let me try to locate it. This code was written by @tzulitai before. It seems the call : ``` $(get_partition_end_offset test-input 1) ``` may get a `null` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667183#comment-16667183 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433904230 Let me try to locate it. This code was written by @tzulitai before. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667164#comment-16667164 ] ASF GitHub Bot commented on FLINK-10600: pnowojski edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433900147 @yanghua but it need to be fixed somehow, otherwise with `set -e` the script will be failing (from time to time?). On the other hand without `set -e` this check might be ignored. Why is it failing only sometimes? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667163#comment-16667163 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433900147 @yanghua but it need to be fixed somehow, otherwise with `set -e` the script will be failing (from time to time?). On the other hand without `set -e` this check might be ignored. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667113#comment-16667113 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433887813 @pnowojski I have fixed some of the issues you mentioned. Regarding the shell error, I don't actually change the logic. I have seen it in my local tests, but not every time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667099#comment-16667099 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433887813 @pnowojski I have fixed some of the issues you mentioned. Regarding the shell error, I don't actually change the code. I didn't have to change this logic. I have seen it in my local tests, but not every time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667027#comment-16667027 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433873778 @yanghua I've created a jira ticket to fix `set -e` in other scripts https://issues.apache.org/jira/browse/FLINK-10711 (that's just FYI) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667018#comment-16667018 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228883766 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -568,6 +585,49 @@ under the License. + + fat-jar-kafka-example + package + + shade + + + false + false + false + + + org.apache.flink.streaming.examples.kafka.KafkaExample + + + KafkaExample + Review comment: It comes from the execution of `fat-jar-kafka-010-example`, I just copied it. Will remove. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1993#comment-1993 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228870191 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -568,6 +585,49 @@ under the License. + + fat-jar-kafka-example + package + + shade + + + false + false + false + + + org.apache.flink.streaming.examples.kafka.KafkaExample + + + KafkaExample + Review comment: Why do we need this comment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1994#comment-1994
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228870018
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -538,6 +550,11 @@ under the License.
shade
+
+
+
org.apache.flink:flink-connector-kafka_${scala.binary.version}
Review comment:
Maybe this one should exclude `0.8`, `0.9` and `0.11` as well for the sake
of consistency?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1995#comment-1995 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228870352 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -568,6 +585,49 @@ under the License. + + fat-jar-kafka-example + package + + shade + + + false + false + false + + + org.apache.flink.streaming.examples.kafka.KafkaExample + + + KafkaExample + + + + * + + org/apache/flink/streaming/examples/kafka/** + org/apache/flink/streaming/** + org/apache/kafka/** + org/apache/curator/** + org/apache/zookeeper/** + org/apache/jute/** + org/I0Itec/** + jline/** + com/yammer/** + kafka/** + + + + + Review comment: ditto: add `0.8` and `0.11`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1996#comment-1996 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228870986 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -568,6 +585,49 @@ under the License. + + fat-jar-kafka-example + package + + shade + + + false + false + false + + + org.apache.flink.streaming.examples.kafka.KafkaExample + + + KafkaExample + + + + * + + org/apache/flink/streaming/examples/kafka/** + org/apache/flink/streaming/** + org/apache/kafka/** + org/apache/curator/** + org/apache/zookeeper/** + org/apache/jute/** + org/I0Itec/** + jline/** + com/yammer/** + kafka/** + + + + Review comment: nit: please reorder `artifactSet` so that both `KafkaExample` and `Kafka10Example` sections are consistent. (either move this to the top, or move `artifactSet` in `Kafka10Example` to the bottom). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665997#comment-16665997 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433603466 @pnowojski I solved the test problem and it is now ready to be reviewed again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665409#comment-16665409 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433472224 No, I mean if we package a jar which contains old connector and new connector and multiple kafka client this pr's end to end test would fail with the same reason. But it seems shade plugin can not pick or exclude specific version of dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665305#comment-16665305 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433449682 @yanghua do you mean this end-to-end test is failing because of https://issues.apache.org/jira/browse/FLINK-10107 and it is blocking this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665123#comment-16665123 ] ASF GitHub Bot commented on FLINK-10600: yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433394209 @pnowojski Using maven-shade-plugin, it's hard to package a jar (in `flink-example-streaming`) that doesn't contain flink kafka 0.10 or old kafka client, which is why FLINK-10107 appears. Unless we adopt the maven profile. But this is not very suitable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665122#comment-16665122 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433394209 @pnowojski Using maven-shade-plugin, it's hard to type a jar that doesn't contain flink kafka 0.10 or old kafka client, which is why FLINK-10107 appears. Unless we adopt the maven profile. But this is not very suitable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665102#comment-16665102
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228505807
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
Yes, I know. My strategy is divided into two steps:
1) First ensure that the end-to-end test passes;
2) Refactor it, still use the original `kafka-common`, and finally there
should be no `modern-kafka-common`
Currently I am working on the first problem (the wrong dependency version
causes the class to not be found).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665094#comment-16665094
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228504540
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
Please deduplicate those two `kafka-common` files. Now this
`setup_confluent_dist` is not only duplicated but also unused.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665058#comment-16665058
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228492647
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
Currently, modern-kafka-common is a simple copy of kafka-common, just to
test as soon as possible. The final modification may be at kafka-common, so I
don't think we need to move it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664951#comment-16664951
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228467388
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
Review comment:
Yes, you are right. I was looking at some outdated code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664945#comment-16664945
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228466189
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
I can not find this anywhere. Is it in the scope of this PR? If not, please
move it to the appropriate PR/commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663891#comment-16663891
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228219980
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+}
+
+function create_kafka_topic {
+ $KAFKA_DIR/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor $1 --partitions $2 --topic $3
+}
+
+function send_messages_to_kafka {
+ echo -e $1 | $KAFKA_DIR/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic $2
+}
+
+function read_messages_from_kafka {
+ $KAFKA_DIR/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning \
+--max-messages $1 \
+--topic $2 \
+--consumer-property group.id=$3 2> /dev/null
+}
+
+function send_messages_to_kafka_avro {
Review comment:
It's used in `test_confluent_schema_registry.sh`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663894#comment-16663894
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228220202
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+}
+
+function create_kafka_topic {
+ $KAFKA_DIR/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor $1 --partitions $2 --topic $3
+}
+
+function send_messages_to_kafka {
+ echo -e $1 | $KAFKA_DIR/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic $2
+}
+
+function read_messages_from_kafka {
+ $KAFKA_DIR/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning \
+--max-messages $1 \
+--topic $2 \
+--consumer-property group.id=$3 2> /dev/null
+}
+
+function send_messages_to_kafka_avro {
+echo -e $1 | $CONFLUENT_DIR/bin/kafka-avro-console-producer --broker-list
localhost:9092 --topic $2 --property value.schema=$3 --property
schema.registry.url=${SCHEMA_REGISTRY_URL}
+}
+
+function modify_num_partitions {
+ $KAFKA_DIR/bin/kafka-topics.sh --alter --topic $1 --partitions $2
--zookeeper localhost:2181
+}
+
+function get_num_partitions {
+ $KAFKA_DIR/bin/kafka-topics.sh --describe --topic $1 --zookeeper
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663889#comment-16663889
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228219566
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
Review comment:
Actually, I did not change the logic of this function. So it also exists in
`kafka-common.sh`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663881#comment-16663881
]
ASF GitHub Bot commented on FLINK-10600:
yanghua commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228217901
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
it used in `test_confluent_schema_registry`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663758#comment-16663758
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228179628
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+}
+
+function create_kafka_topic {
+ $KAFKA_DIR/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor $1 --partitions $2 --topic $3
+}
+
+function send_messages_to_kafka {
+ echo -e $1 | $KAFKA_DIR/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic $2
+}
+
+function read_messages_from_kafka {
+ $KAFKA_DIR/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning \
+--max-messages $1 \
+--topic $2 \
+--consumer-property group.id=$3 2> /dev/null
+}
+
+function send_messages_to_kafka_avro {
+echo -e $1 | $CONFLUENT_DIR/bin/kafka-avro-console-producer --broker-list
localhost:9092 --topic $2 --property value.schema=$3 --property
schema.registry.url=${SCHEMA_REGISTRY_URL}
+}
+
+function modify_num_partitions {
+ $KAFKA_DIR/bin/kafka-topics.sh --alter --topic $1 --partitions $2
--zookeeper localhost:2181
+}
+
+function get_num_partitions {
+ $KAFKA_DIR/bin/kafka-topics.sh --describe --topic $1 --zookeeper
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663757#comment-16663757
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228179104
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
Review comment:
Why the script for kafka 0.10 didn't have this killing part? Should it have?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663756#comment-16663756
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228181381
##
File path:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/KafkaExample.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.kafka;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
+
+/**
+ * A simple example that shows how to read from and write to modern Kafka.
This will read String messages
+ * from the input topic, parse them into a POJO type {@link KafkaEvent}, group
by some key, and finally
+ * perform a rolling addition on each key for which the results are written
back to another topic.
+ *
+ * This example also demonstrates using a watermark assigner to generate
per-partition
+ * watermarks directly in the Flink Kafka consumer. For demonstration
purposes, it is assumed that
+ * the String messages are of formatted as a (word,frequency,timestamp) tuple.
+ *
+ * Example usage:
+ * --input-topic test-input --output-topic test-output --bootstrap.servers
localhost:9092
+ * --zookeeper.connect localhost:2181 --group.id myconsumer
+ */
+public class KafkaExample {
+
+ public static void main(String[] args) throws Exception {
+ // parse input arguments
+ final ParameterTool parameterTool =
ParameterTool.fromArgs(args);
+
+ if (parameterTool.getNumberOfParameters() < 5) {
+ System.out.println("Missing parameters!\n" +
+ "Usage: Kafka --input-topic
--output-topic " +
+ "--bootstrap.servers " +
+ "--zookeeper.connect --group.id
");
+ return;
+ }
+
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ env.getConfig().disableSysoutLogging();
+
env.getConfig().setRestartStrategy(RestartStrategies.fixedDelayRestart(4,
1));
+ env.enableCheckpointing(5000); // create a checkpoint every 5
seconds
+ env.getConfig().setGlobalJobParameters(parameterTool); // make
parameters available in the web interface
+ env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+
+ DataStream input = env
+ .addSource(
+ new FlinkKafkaConsumer<>(
+
parameterTool.getRequired("input-topic"),
+ new KafkaEventSchema(),
+ parameterTool.getProperties())
+ .assignTimestampsAndWatermarks(new
CustomWatermarkExtractor()))
+ .keyBy("word")
+ .map(new RollingAdditionMapper());
+
+ input.addSink(
+ new FlinkKafkaProducer<>(
+ parameterTool.getRequired("output-topic"),
+ new KafkaEventSchema(),
+ parameterTool.getProperties()));
Review comment:
I think it would be better to use `EXACTLY_ONCE` mode here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663760#comment-16663760
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228178672
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
Review comment:
is this used anywhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663754#comment-16663754
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228178423
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
Review comment:
for example it looks like this method differs only with the `KAFKA_URL`
"parameter"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663759#comment-16663759
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228179323
##
File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh
##
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -o pipefail
+
+if [[ -z $TEST_DATA_DIR ]]; then
+ echo "Must run common.sh before modern-kafka-common.sh."
+ exit 1
+fi
+
+KAFKA_DIR=$TEST_DATA_DIR/kafka_2.11-2.0.0
+CONFLUENT_DIR=$TEST_DATA_DIR/confluent-5.0.0
+SCHEMA_REGISTRY_PORT=8082
+SCHEMA_REGISTRY_URL=http://localhost:${SCHEMA_REGISTRY_PORT}
+
+function setup_kafka_dist {
+ # download Kafka
+ mkdir -p $TEST_DATA_DIR
+ KAFKA_URL="https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz;
+ echo "Downloading Kafka from $KAFKA_URL"
+ curl "$KAFKA_URL" > $TEST_DATA_DIR/modern-kafka.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-kafka.tgz -C $TEST_DATA_DIR/
+
+ # fix kafka config
+ sed -i -e "s+^\(dataDir\s*=\s*\).*$+\1$TEST_DATA_DIR/zookeeper+"
$KAFKA_DIR/config/zookeeper.properties
+ sed -i -e "s+^\(log\.dirs\s*=\s*\).*$+\1$TEST_DATA_DIR/kafka+"
$KAFKA_DIR/config/server.properties
+}
+
+function setup_confluent_dist {
+ # download confluent
+ mkdir -p $TEST_DATA_DIR
+
CONFLUENT_URL="http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz;
+ echo "Downloading confluent from $CONFLUENT_URL"
+ curl "$CONFLUENT_URL" > $TEST_DATA_DIR/modern-confluent.tgz
+
+ tar xzf $TEST_DATA_DIR/modern-confluent.tgz -C $TEST_DATA_DIR/
+
+ # fix confluent config
+ sed -i -e
"s#listeners=http://0.0.0.0:8081#listeners=http://0.0.0.0:${SCHEMA_REGISTRY_PORT}#;
$CONFLUENT_DIR/etc/schema-registry/schema-registry.properties
+}
+
+function start_kafka_cluster {
+ if [[ -z $KAFKA_DIR ]]; then
+echo "Must run 'setup_kafka_dist' before attempting to start Kafka cluster"
+exit 1
+ fi
+
+ $KAFKA_DIR/bin/zookeeper-server-start.sh -daemon
$KAFKA_DIR/config/zookeeper.properties
+ $KAFKA_DIR/bin/kafka-server-start.sh -daemon
$KAFKA_DIR/config/server.properties
+
+ # zookeeper outputs the "Node does not exist" bit to stderr
+ while [[ $($KAFKA_DIR/bin/zookeeper-shell.sh localhost:2181 get
/brokers/ids/0 2>&1) =~ .*Node\ does\ not\ exist.* ]]; do
+echo "Waiting for broker..."
+sleep 1
+ done
+}
+
+function stop_kafka_cluster {
+ $KAFKA_DIR/bin/kafka-server-stop.sh
+ $KAFKA_DIR/bin/zookeeper-server-stop.sh
+
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+
+ if [ ! -z "$PIDS" ]; then
+kill -s TERM $PIDS
+ fi
+}
+
+function create_kafka_topic {
+ $KAFKA_DIR/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor $1 --partitions $2 --topic $3
+}
+
+function send_messages_to_kafka {
+ echo -e $1 | $KAFKA_DIR/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic $2
+}
+
+function read_messages_from_kafka {
+ $KAFKA_DIR/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning \
+--max-messages $1 \
+--topic $2 \
+--consumer-property group.id=$3 2> /dev/null
+}
+
+function send_messages_to_kafka_avro {
Review comment:
Is this used anywhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
>
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663752#comment-16663752 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228179894 ## File path: flink-end-to-end-tests/test-scripts/test_streaming_kafka.sh ## @@ -0,0 +1,114 @@ +#!/usr/bin/env bash Review comment: Please deduplicate this as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663755#comment-16663755
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r228181171
##
File path:
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/kafka/KafkaExample.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.examples.kafka;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
+
+/**
+ * A simple example that shows how to read from and write to modern Kafka.
This will read String messages
+ * from the input topic, parse them into a POJO type {@link KafkaEvent}, group
by some key, and finally
+ * perform a rolling addition on each key for which the results are written
back to another topic.
+ *
+ * This example also demonstrates using a watermark assigner to generate
per-partition
+ * watermarks directly in the Flink Kafka consumer. For demonstration
purposes, it is assumed that
+ * the String messages are of formatted as a (word,frequency,timestamp) tuple.
+ *
+ * Example usage:
+ * --input-topic test-input --output-topic test-output --bootstrap.servers
localhost:9092
+ * --zookeeper.connect localhost:2181 --group.id myconsumer
+ */
+public class KafkaExample {
+
+ public static void main(String[] args) throws Exception {
Review comment:
lines from 43 to 60 also could be deduplicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663753#comment-16663753 ] ASF GitHub Bot commented on FLINK-10600: pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#discussion_r228178228 ## File path: flink-end-to-end-tests/test-scripts/test-data/modern-kafka-common.sh ## @@ -0,0 +1,152 @@ +#!/usr/bin/env bash Review comment: Please deduplicate the code here with `kafka-common.sh` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663589#comment-16663589 ] ASF GitHub Bot commented on FLINK-10600: yanghua opened a new pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924 ## What is the purpose of the change *This pull request provides End-to-end test cases for modern Kafka connectors* ## Brief change log - *Provide End-to-end test cases for modern Kafka connectors* ## Verifying this change This change added tests and can be verified as follows: - *Added end-to-end integration tests for modern kafka connector * ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663585#comment-16663585 ] ASF GitHub Bot commented on FLINK-10600: yanghua closed pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663458#comment-16663458 ] ASF GitHub Bot commented on FLINK-10600: zentol commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-432969415 ```git update-index --chmod=+x ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663432#comment-16663432 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-432962337 @pnowojski The error log shows my new shell script has permission problem : `test_streaming_kafka.sh: Permission denied`. Where can I give it execution permission? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663102#comment-16663102 ] ASF GitHub Bot commented on FLINK-10600: yanghua opened a new pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924 ## What is the purpose of the change *This pull request provides End-to-end test cases for modern Kafka connectors* ## Brief change log - *Provide End-to-end test cases for modern Kafka connectors* ## Verifying this change This change added tests and can be verified as follows: - *Added end-to-end integration tests for modern kafka connector * ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662173#comment-16662173 ] Piotr Nowojski commented on FLINK-10600: Great :) Please let me know once you have PR ready. > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662150#comment-16662150 ] vinoyang commented on FLINK-10600: -- [~pnowojski] Yes, I am doing it, I will try to implement it as soon as possible. > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
