[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673196#comment-16673196
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys closed pull request #7001: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7001
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/splits/split_ha.sh b/splits/split_ha.sh
index 7a8d8ec8a25..628b716d7cb 100755
--- a/splits/split_ha.sh
+++ b/splits/split_ha.sh
@@ -50,5 +50,10 @@ run_test "Running HA (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts
run_test "Running HA (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true false"
"skip_check_exceptions"
run_test "Running HA (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true true"
"skip_check_exceptions"
+run_test "Running HA per-job cluster (file, async) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file false
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
true" "skip_check_exceptions"
+
printf "\n[PASS] All tests passed\n"
exit 0
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673197#comment-16673197
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys closed pull request #7002: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7002
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/splits/split_ha.sh b/splits/split_ha.sh
index 7a8d8ec8a25..628b716d7cb 100755
--- a/splits/split_ha.sh
+++ b/splits/split_ha.sh
@@ -50,5 +50,10 @@ run_test "Running HA (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts
run_test "Running HA (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true false"
"skip_check_exceptions"
run_test "Running HA (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true true"
"skip_check_exceptions"
+run_test "Running HA per-job cluster (file, async) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file false
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
true" "skip_check_exceptions"
+
printf "\n[PASS] All tests passed\n"
exit 0
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673195#comment-16673195
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys closed pull request #7000: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7000
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/splits/split_ha.sh b/splits/split_ha.sh
index 7a8d8ec8a25..628b716d7cb 100755
--- a/splits/split_ha.sh
+++ b/splits/split_ha.sh
@@ -50,5 +50,10 @@ run_test "Running HA (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts
run_test "Running HA (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true false"
"skip_check_exceptions"
run_test "Running HA (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true true"
"skip_check_exceptions"
+run_test "Running HA per-job cluster (file, async) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file false
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
true" "skip_check_exceptions"
+
printf "\n[PASS] All tests passed\n"
exit 0
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672767#comment-16672767
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys opened a new pull request #7002: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7002
Enabled e2e tests in cron job
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672764#comment-16672764
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys opened a new pull request #7001: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7001
Enabled e2e tests in cron job
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672762#comment-16672762
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys opened a new pull request #7000: [FLINK-10632] Enabled running general
purpose testing job with failure in per-job cluster
URL: https://github.com/apache/flink/pull/7000
Enabled e2e tests in cron job
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672761#comment-16672761
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys closed pull request #6943: [FLINK-10632][e2e] Running general purpose
testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-end-to-end-tests/run-nightly-tests.sh
b/flink-end-to-end-tests/run-nightly-tests.sh
index bbb0e9e5885..6a6c8b6729b 100755
--- a/flink-end-to-end-tests/run-nightly-tests.sh
+++ b/flink-end-to-end-tests/run-nightly-tests.sh
@@ -61,6 +61,11 @@ run_test "Running HA (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts
run_test "Running HA (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true false"
"skip_check_exceptions"
run_test "Running HA (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_datastream.sh rocks true true"
"skip_check_exceptions"
+run_test "Running HA per-job cluster (file, async) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (file, sync) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh file false
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, non-incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
false" "skip_check_exceptions"
+run_test "Running HA per-job cluster (rocks, incremental) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_ha_per_job_cluster_datastream.sh rocks true
true" "skip_check_exceptions"
+
run_test "Resuming Savepoint (file, async, no parallelism change) end-to-end
test" "$END_TO_END_DIR/test-scripts/test_resume_savepoint.sh 2 2 file true"
run_test "Resuming Savepoint (file, sync, no parallelism change) end-to-end
test" "$END_TO_END_DIR/test-scripts/test_resume_savepoint.sh 2 2 file false"
run_test "Resuming Savepoint (file, async, scale up) end-to-end test"
"$END_TO_END_DIR/test-scripts/test_resume_savepoint.sh 2 4 file true"
diff --git a/flink-end-to-end-tests/test-scripts/common.sh
b/flink-end-to-end-tests/test-scripts/common.sh
index e815a85afdd..09c42af8742 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -33,6 +33,7 @@ case "$(uname -s)" in
esac
export EXIT_CODE=0
+export TASK_SLOTS_PER_TM_HA=4
echo "Flink dist directory: $FLINK_DIR"
@@ -123,7 +124,7 @@ function create_ha_config() {
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 1024
taskmanager.heap.mb: 1024
-taskmanager.numberOfTaskSlots: 4
+taskmanager.numberOfTaskSlots: ${TASK_SLOTS_PER_TM_HA}
#==
# High Availability
@@ -238,9 +239,7 @@ function start_local_zk {
done < <(grep "^server\." "${FLINK_DIR}/conf/zoo.cfg")
}
-function start_cluster {
- "$FLINK_DIR"/bin/start-cluster.sh
-
+function wait_dispatcher_running {
# wait at most 10 seconds until the dispatcher is up
local QUERY_URL
if [ "x$USE_SSL" = "xON" ]; then
@@ -264,6 +263,11 @@ function start_cluster {
done
}
+function start_cluster {
+ "$FLINK_DIR"/bin/start-cluster.sh
+ wait_dispatcher_running
+}
+
function start_taskmanagers {
tmnum=$1
echo "Start ${tmnum} more task managers"
@@ -599,6 +603,33 @@ function wait_oper_metric_num_in_records {
done
}
+function wait_num_of_occurence_in_logs {
+local text=$1
+local number=$2
+local logs
+if [ -z "$3" ]; then
+logs="standalonesession"
+else
+logs="$3"
+fi
+
+echo "Waiting for text ${text} to appear ${number} of times in logs..."
+
+while : ; do
+ N=$(grep -o "${text}" $FLINK_DIR/log/*${logs}*.log | wc -l)
+
+ if [ -z $N ]; then
+N=0
+ fi
+
+ if (( N < number )); then
+sleep 1
+ else
+break
+ fi
+done
+}
+
function wait_num_checkpoints {
JOB=$1
NUM_CHECKPOINTS=$2
diff --git a/flink-end-to-end-tests/test-scripts/common_ha.sh
b/flink-end-to-end-tests/test-scripts/common_ha.sh
index f476bac5e66..6521cb40127 100644
--- a/flink-end-to-end-tests/test-scripts/common_ha.sh
+++ b/flink-end-to-end-tests/test-scripts/common_ha.sh
@@ -24,7 +24,7 @@ CLEARED=0
JM_WATCHDOG_PID=0
TM_WATCHDOG_PID=0
-function stop_cluster_and_watchdog() {
+function stop_watchdogs() {
if [ ${CLEARED} -eq 0 ]; then
if ! [ ${JM_WATCH
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669911#comment-16669911
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys commented on issue #6943: [FLINK-10632][e2e] Running general purpose
testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#issuecomment-434640033
I've tried to address your comments @tillrohrmann. I would appreciate if you
have had another look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669910#comment-16669910
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229641650
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
Review comment:
I've switched to the basic regular expressions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669905#comment-16669905
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229639535
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
+
+function run_ha_test() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local JM_KILLS=3
+
+
+CLEARED=0
+
+# add job jar to cluster classpath
+cp ${TEST_PROGRAM_JAR} ${FLINK_LIB_DIR}
+
+# start the cluster on HA mode
+create_ha_config
+start_local_zk
+
+echo "Running o
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669880#comment-16669880
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229632726
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
Review comment:
It is similar, but it performs different checks. Looks for different texts
and number of them. I extracted the comparing number of occurrence logic but
don't think there is much more that can be unified.
This is an automated message from the Apache Git Servic
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668967#comment-16668967
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229352991
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
Review comment:
Is this really needed? I think that setting `environment.parallelism` should
already be enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668972#comment-16668972
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229343984
##
File path: flink-end-to-end-tests/test-scripts/test_ha_datastream.sh
##
@@ -22,6 +22,20 @@ source "$(dirname "$0")"/common_ha.sh
TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/DataStreamAllroundTestProgram.jar
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ # kill the cluster and zookeeper
+ stop_watchdogs
+ shutdown_all
Review comment:
Same here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668974#comment-16668974
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229353255
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
+
+function run_ha_test() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local JM_KILLS=3
+
+
+CLEARED=0
+
+# add job jar to cluster classpath
+cp ${TEST_PROGRAM_JAR} ${FLINK_LIB_DIR}
+
+# start the cluster on HA mode
+create_ha_config
+start_local_zk
+
+echo "Runni
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668969#comment-16668969
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229346067
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
+
+function run_ha_test() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local JM_KILLS=3
+
+
+CLEARED=0
+
+# add job jar to cluster classpath
+cp ${TEST_PROGRAM_JAR} ${FLINK_LIB_DIR}
+
+# start the cluster on HA mode
+create_ha_config
+start_local_zk
+
+echo "Runni
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668968#comment-16668968
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229344187
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
Review comment:
No need to call this, since the test runner will call it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668975#comment-16668975
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229353456
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
+
+function run_ha_test() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local JM_KILLS=3
+
+
+CLEARED=0
+
+# add job jar to cluster classpath
+cp ${TEST_PROGRAM_JAR} ${FLINK_LIB_DIR}
+
+# start the cluster on HA mode
+create_ha_config
+start_local_zk
+
+echo "Runni
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668971#comment-16668971
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229358923
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
Review comment:
This looks like a duplicate of `common_ha.sh#verify_logs`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, pleas
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668973#comment-16668973
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229376944
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
Review comment:
The standard `grep` application on Mac OS does not support `-P` because it's
FreeBSD's grep version. On Mac OS you can pass the `-E` option to enable
extended regex support which would support `\d+`. I think we should make it
Unix compatible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affect
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668966#comment-16668966
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229343747
##
File path: flink-end-to-end-tests/test-scripts/test_ha_dataset.sh
##
@@ -22,6 +22,20 @@ source "$(dirname "$0")"/common_ha.sh
TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-dataset-allround-test/target/DataSetAllroundTestProgram.jar
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ # kill the cluster and zookeeper
+ stop_watchdogs
+ shutdown_all
Review comment:
I think `shutdown_all` will be called by the test runner in
`test-runner-common.sh`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668970#comment-16668970
]
ASF GitHub Bot commented on FLINK-10632:
tillrohrmann commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229353357
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
+
+function run_ha_test() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local JM_KILLS=3
+
+
+CLEARED=0
+
+# add job jar to cluster classpath
+cp ${TEST_PROGRAM_JAR} ${FLINK_LIB_DIR}
+
+# start the cluster on HA mode
+create_ha_config
+start_local_zk
+
+echo "Runni
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665265#comment-16665265
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys opened a new pull request #6943: [FLINK-10632][e2e] Running general
purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943
## What is the purpose of the change
Add test that runs general purpose job with failure in per-job cluster mode.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs /
JavaDocs / not documented)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Run general purpose test job with failures in per-job mode
> --
>
> Key: FLINK-10632
> URL: https://issues.apache.org/jira/browse/FLINK-10632
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Similar to FLINK-8973, we should add an end-to-end which runs the general
> datastream job with failures on a per-job cluster with HA enabled (either
> directly the {{StandaloneJobClusterEntrypoint}} or a docker image based on
> this entrypoint).
> We should kill the TMs as well as the cluster entrypoint and verify that the
> job recovers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
