[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412292#comment-16412292 ] ASF GitHub Bot commented on FLINK-9067: --- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/5759 [FLINK-9067] [e2eTests] Add StreamSQLTestProgram and test run script. ## What is the purpose of the change * Adds an end-to-end test for a streaming SQL query * The query fails at the first execution attempt, recovers, and completes in the second attempt ## Brief change log * Add a Streaming SQL Test Job * Add a script to setup the Flink cluster, run the job, stop the cluster, and verify the result * Register the script in the nightly test jobs ## Verifying this change * This PR adds a test. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **n/a** You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink e2eTestSQLStream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5759.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5759 commit 7a14340827f5a31a11c71e05075732d3c455cda5 Author: Fabian Hueske Date: 2018-03-23T20:42:53Z [FLINK-9067] [e2eTests] Add StreamSQLTestProgram and test run script. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412543#comment-16412543 ] ASF GitHub Bot commented on FLINK-9067: --- Github user zhangminglei commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r176906704 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib --- End diff -- Hi, @fhueske I have some small questions here. I can see that there are some jar files in the example folder(```$FLINK_DIR/example```), some jar files in the ```$FLINK_DIR/lib``` folder, and other jar files in the ```$FLINK_DIR/opt``` folder. Why did you copy this jar file from opt folder to lib folder, instead of putting to ```$FLINK_DIR/example``` ? Thanks. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412546#comment-16412546 ] ASF GitHub Bot commented on FLINK-9067: --- Github user zhangminglei commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r176906809 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib --- End diff -- Oh. I think because you start taskmanager.sh, so you have to copy that. Not sure about it. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413713#comment-16413713 ] ASF GitHub Bot commented on FLINK-9067: --- Github user twalthr commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177058370 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib --- End diff -- @zhangminglei there is a known issue with a classloader that can not be removed. See the notice in https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/#setup. That's why the optional Table API jar is moved to lib. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413883#comment-16413883 ] ASF GitHub Bot commented on FLINK-9067: --- Github user zhangminglei commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177099106 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib --- End diff -- Thanks @twalthr . I will take a look. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415582#comment-16415582 ] ASF GitHub Bot commented on FLINK-9067: --- Github user twalthr commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177416632 --- Diff: flink-end-to-end-tests/flink-stream-sql-test/pom.xml --- @@ -0,0 +1,126 @@ + + +http://maven.apache.org/POM/4.0.0"; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + + + flink-end-to-end-tests + org.apache.flink + 1.6-SNAPSHOT + .. + + + 4.0.0 + + flink-stream-sql-test_${scala.binary.version} + flink-stream-sql-test + jar + + + + org.apache.flink + flink-core + ${project.version} + provided + + + org.apache.flink + flink-streaming-scala_${scala.binary.version} + ${project.version} + provided + + + org.apache.flink + flink-table_${scala.binary.version} + ${project.version} + provided + + + org.apache.flink + flink-connector-filesystem_${scala.binary.version} + ${project.version} + + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.0.0 + + + package + + shade + + + + + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + org.apache.flink.sql.tests.StreamSQLTestProgram + + + + + + + + --- End diff -- This can be simiplified similar to the dataset allround test. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415583#comment-16415583 ] ASF GitHub Bot commented on FLINK-9067: --- Github user twalthr commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177416467 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib + +start_cluster +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start + +$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath $TEST_DATA_DIR/out/result + +stop_cluster +$FLINK_DIR/bin/taskmanager.sh stop-all + +# remove flink-table from lib folder +rm $FLINK_DIR/lib/flink-table*jar --- End diff -- We should convert this into a function and call `trap` to make sure this is always executed for clean up. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415581#comment-16415581 ] ASF GitHub Bot commented on FLINK-9067: --- Github user twalthr commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177416248 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib + +start_cluster +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start + +$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath $TEST_DATA_DIR/out/result + +stop_cluster +$FLINK_DIR/bin/taskmanager.sh stop-all + +# remove flink-table from lib folder +rm $FLINK_DIR/lib/flink-table*jar + +# collect results from files +cat /tmp/xxx/part-0-0 /tmp/xxx/_part-0-1.pending > $TEST_DATA_DIR/out/result-complete --- End diff -- I guess this line is not correct ;) > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415653#comment-16415653 ] ASF GitHub Bot commented on FLINK-9067: --- Github user zhangminglei commented on a diff in the pull request: https://github.com/apache/flink/pull/5759#discussion_r177433238 --- Diff: flink-end-to-end-tests/test-scripts/test_streaming_sql.sh --- @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +source "$(dirname "$0")"/common.sh + +TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/flink-stream-sql-test/target/StreamSQLTestProgram.jar + +# copy flink-table jar into lib folder +cp $FLINK_DIR/opt/flink-table*jar $FLINK_DIR/lib + +start_cluster +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start +$FLINK_DIR/bin/taskmanager.sh start + +$FLINK_DIR/bin/flink run -p 4 $TEST_PROGRAM_JAR -outputPath $TEST_DATA_DIR/out/result + +stop_cluster +$FLINK_DIR/bin/taskmanager.sh stop-all + +# remove flink-table from lib folder +rm $FLINK_DIR/lib/flink-table*jar --- End diff -- Exactly. > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9067) End-to-end test: Stream SQL query with failure and retry
[ https://issues.apache.org/jira/browse/FLINK-9067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415657#comment-16415657 ] ASF GitHub Bot commented on FLINK-9067: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5759 > End-to-end test: Stream SQL query with failure and retry > > > Key: FLINK-9067 > URL: https://issues.apache.org/jira/browse/FLINK-9067 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL, Tests >Affects Versions: 1.5.0 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Major > > Implement a test job that runs a streaming SQL query with a temporary failure > and recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005)