(flink-connector-pulsar) branch main updated: [hotfix] Update copyright year to 2024
This is an automated email from the ASF dual-hosted git repository. leonard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git The following commit(s) were added to refs/heads/main by this push: new 5889eb2 [hotfix] Update copyright year to 2024 5889eb2 is described below commit 5889eb21d13126f0b4e616cabb6f07365188215e Author: Hang Ruan AuthorDate: Wed Jan 31 09:51:57 2024 +0800 [hotfix] Update copyright year to 2024 This closes #82. --- flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE b/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE index 4730a99..de3d655 100644 --- a/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE +++ b/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE @@ -1,5 +1,5 @@ flink-sql-connector-pulsar -Copyright 2014-2023 The Apache Software Foundation +Copyright 2014-2024 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
(flink-connector-hbase) branch main updated (42870d5 -> 91d166d)
This is an automated email from the ASF dual-hosted git repository. leonard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-hbase.git from 42870d5 [hotfix] Remove 1.19-SNAPSHOT since HBase currently can't compile for that version add 91d166d [hotfix] Update copyright year to 2024 No new revisions were added by this update. Summary of changes: flink-sql-connector-hbase-1.4/src/main/resources/META-INF/NOTICE | 2 +- flink-sql-connector-hbase-2.2/src/main/resources/META-INF/NOTICE | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(flink-connector-gcp-pubsub) branch main updated: [hotfix][ci] Make CI against multiple Flink versions
This is an automated email from the ASF dual-hosted git repository. leonard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git The following commit(s) were added to refs/heads/main by this push: new 4c155b4 [hotfix][ci] Make CI against multiple Flink versions 4c155b4 is described below commit 4c155b49e41d157d780a2d1e80c388017b0a2941 Author: Hang Ruan AuthorDate: Wed Jan 31 11:49:47 2024 +0800 [hotfix][ci] Make CI against multiple Flink versions This closes #21. --- .github/workflows/push_pr.yml | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml index 335f108..99cf8d7 100644 --- a/.github/workflows/push_pr.yml +++ b/.github/workflows/push_pr.yml @@ -23,6 +23,9 @@ concurrency: cancel-in-progress: true jobs: compile_and_test: +strategy: + matrix: +flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT, 1.18-SNAPSHOT] uses: apache/flink-connector-shared-utils/.github/workflows/ci.yml@ci_utils with: - flink_version: 1.17.0 + flink_version: ${{ matrix.flink }}
(flink-connector-elasticsearch) branch main updated (153b8fc -> 4914929)
This is an automated email from the ASF dual-hosted git repository. leonard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-elasticsearch.git from 153b8fc [hotfix] Set maximum supported Flink version to 1.17 for `v3.0` add 7bda67e [FLINK-34113][elasticsearch] Make flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces This closes #88. add 4914929 [hotfix] Add dependency convergence for org.yaml:snakeyaml No new revisions were added by this update. Summary of changes: flink-connector-elasticsearch-base/pom.xml | 8 ++ .../sink/ElasticsearchSinkBuilderBaseTest.java | 100 + flink-connector-elasticsearch6/pom.xml | 8 ++ flink-connector-elasticsearch7/pom.xml | 8 ++ .../src/main/resources/META-INF/NOTICE | 2 + pom.xml| 7 ++ 6 files changed, 36 insertions(+), 97 deletions(-)
(flink-connector-elasticsearch) branch dependabot/maven/org.yaml-snakeyaml-2.0 created (now 60c726d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.yaml-snakeyaml-2.0 in repository https://gitbox.apache.org/repos/asf/flink-connector-elasticsearch.git at 60c726d Bump org.yaml:snakeyaml from 1.31 to 2.0 No new revisions were added by this update.
(flink-connector-jdbc) branch main updated (5a90eb0a -> 611b45cf)
This is an automated email from the ASF dual-hosted git repository. guoweijie pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-jdbc.git from 5a90eb0a [FLINK-33365] include filters with Lookup joins add 611b45cf [FLINK-34259] Creating ExecutionConfig instead of mocking it in tests No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/jdbc/JdbcTestBase.java | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
(flink) branch master updated (0ba3e76b147 -> b5a2ee4e988)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 0ba3e76b147 [docs] Fix typo in ide_setup.md (#20877) add b5a2ee4e988 [FLINK-34286][k8s] Attach cluster config map labels at creation time No new revisions were added by this update. Summary of changes: .../KubernetesLeaderElectionDriver.java| 10 -- .../resources/KubernetesLeaderElector.java | 16 +++-- .../KubernetesLeaderElectionDriverTest.java| 23 .../resources/KubernetesLeaderElectorITCase.java | 42 ++ 4 files changed, 56 insertions(+), 35 deletions(-)
(flink) branch master updated (b5a2ee4e988 -> 85edd784fc7)
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from b5a2ee4e988 [FLINK-34286][k8s] Attach cluster config map labels at creation time new 1f23e2ce6f1 [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found new f10702b8c23 [hotfix][ci] Fixes condition for filtering NOTICE files new adde027cae6 [hotfix][build] Adds Maven and Java version to Maven command new edfdc89a029 [hotfix][ci] Replaces spaces with tabs new e4b4718009a [hotfix][ci] Adds proper log message at the end of a successful CI run new f4c0b716b01 [FLINK-34062][build] Adds JDK 21 surefire configuration. new a810088688c [FLINK-27082][ci] Updates github-actions profile. new 85edd784fc7 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow The 8 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/actions/job_init/action.yml| 72 .github/actions/run_mvn/action.yml | 42 +++ .../workflows/ci.yml | 56 ++- .github/workflows/template.flink-ci.yml| 386 + .github/workflows/template.pre-compile-checks.yml | 77 pom.xml| 51 ++- tools/azure-pipelines/create_build_artifact.sh | 12 +- tools/azure-pipelines/debug_files_utils.sh | 62 ++-- tools/azure-pipelines/unpack_build_artifact.sh | 20 +- tools/azure-pipelines/uploading_watchdog.sh| 4 +- .../tools/ci/licensecheck/NoticeFileChecker.java | 3 +- tools/ci/maven-utils.sh| 1 + 12 files changed, 716 insertions(+), 70 deletions(-) create mode 100644 .github/actions/job_init/action.yml create mode 100644 .github/actions/run_mvn/action.yml copy tools/releasing/deploy_staging_jars.sh => .github/workflows/ci.yml (52%) mode change 100755 => 100644 create mode 100644 .github/workflows/template.flink-ci.yml create mode 100644 .github/workflows/template.pre-compile-checks.yml
(flink) 03/08: [hotfix][build] Adds Maven and Java version to Maven command
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit adde027cae657909d4e4cc6411f6b65e67970c1b Author: Matthias Pohl AuthorDate: Tue Jan 16 16:20:45 2024 +0100 [hotfix][build] Adds Maven and Java version to Maven command --- tools/ci/maven-utils.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/ci/maven-utils.sh b/tools/ci/maven-utils.sh index 2ac16179b23..ab535af4320 100755 --- a/tools/ci/maven-utils.sh +++ b/tools/ci/maven-utils.sh @@ -17,6 +17,7 @@ # Utility for invoking Maven in CI function run_mvn { if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then + $MAVEN_WRAPPER --version echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'" fi $MAVEN_WRAPPER $MVN_GLOBAL_OPTIONS "${@}"
(flink) 01/08: [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit 1f23e2ce6f1ef9295d6899034f005467d8c94607 Author: Matthias Pohl AuthorDate: Thu Dec 21 18:27:46 2023 +0100 [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found --- tools/azure-pipelines/create_build_artifact.sh | 2 +- tools/azure-pipelines/unpack_build_artifact.sh | 11 +-- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/tools/azure-pipelines/create_build_artifact.sh b/tools/azure-pipelines/create_build_artifact.sh index 809d14fe651..0beeca3b075 100755 --- a/tools/azure-pipelines/create_build_artifact.sh +++ b/tools/azure-pipelines/create_build_artifact.sh @@ -28,7 +28,7 @@ echo "Minimizing artifact files" # by removing files not required for subsequent stages # jars are re-built in subsequent stages, so no need to cache them (cannot be avoided) -find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' | xargs rm -rf +find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' -exec rm -rf {} \; # .git directory # not deleting this can cause build stability issues diff --git a/tools/azure-pipelines/unpack_build_artifact.sh b/tools/azure-pipelines/unpack_build_artifact.sh index 15d5c7c1057..68f8d221aae 100755 --- a/tools/azure-pipelines/unpack_build_artifact.sh +++ b/tools/azure-pipelines/unpack_build_artifact.sh @@ -28,15 +28,14 @@ cp -RT "$FLINK_ARTIFACT_DIR" "." echo "Adjusting timestamps" # adjust timestamps of proto file to avoid re-generation -find . -type f -name '*.proto' | xargs touch +find . -type f -name '*.proto' -exec touch {} \; # wait a bit for better odds of different timestamps sleep 5 # adjust timestamps to prevent recompilation -find . -type f -name '*.java' | xargs touch -find . -type f -name '*.scala' | xargs touch +find . -type f -name '*.java' -exec touch {} \; +find . -type f -name '*.scala' -exec touch {} \; # wait a bit for better odds of different timestamps sleep 5 -find . -type f -name '*.class' | xargs touch -find . -type f -name '*.timestamp' | xargs touch - +find . -type f -name '*.class' -exec touch {} \; +find . -type f -name '*.timestamp' -exec touch {} \;
(flink) 08/08: [FLINK-33914][ci] Introduces a basic GitHub Actions workflow
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit 85edd784fc72c1784849e2b122cbf3215f89817c Author: Matthias Pohl AuthorDate: Sun Dec 10 13:36:57 2023 +0100 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow - Adds basic CI workflow - Adds composite workflow for starting Flink's CI stages - Updates CI helper scripts to support GitHub Actions - Adds composite workflow for doing pre-compile checks - Adds custom action to enable maven-utils.sh in workflows - Adds custom action for initializing a job --- .github/actions/job_init/action.yml | 72 .github/actions/run_mvn/action.yml| 42 +++ .github/workflows/ci.yml | 40 +++ .github/workflows/template.flink-ci.yml | 386 ++ .github/workflows/template.pre-compile-checks.yml | 77 + tools/azure-pipelines/create_build_artifact.sh| 10 +- tools/azure-pipelines/debug_files_utils.sh| 18 + tools/azure-pipelines/unpack_build_artifact.sh| 9 +- tools/azure-pipelines/uploading_watchdog.sh | 2 + 9 files changed, 653 insertions(+), 3 deletions(-) diff --git a/.github/actions/job_init/action.yml b/.github/actions/job_init/action.yml new file mode 100644 index 000..5158d855857 --- /dev/null +++ b/.github/actions/job_init/action.yml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Initializes the job" +description: "Does all the necessary steps to set up the job" +inputs: + jdk_version: +description: "The JDK version that's supposed to be used." +required: true + maven_repo_folder: +description: "The location of the local Maven repository (not setting this parameter will omit the caching of Maven artifacts)." +required: false +default: "" + source_directory: +description: "Specifies the directory from which the code should be moved from (needed for containerized runs; not setting this parameter will omit moving the checkout)." +required: false +default: "" + target_directory: +description: "Specifies the directory to which the code should be moved to (needed for containerized runs; not setting this parameter will omit moving the checkout)." +required: false +default: "" +runs: + using: "composite" + steps: +- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time" + shell: bash + run: | +job_start_time="$(date --rfc-3339=seconds)" +echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}" +echo "The job's start time is set to ${job_start_time}." + +- name: "Set JDK version to ${{ inputs.jdk_version }}" + shell: bash + run: | +echo "JAVA_HOME=$JAVA_HOME_${{ inputs.jdk_version }}_X64" >> "${GITHUB_ENV}" +echo "PATH=$JAVA_HOME_${{ inputs.jdk_version }}_X64/bin:$PATH" >> "${GITHUB_ENV}" + +- name: "Setup Maven package cache" + if: ${{ inputs.maven_repo_folder != '' }} + uses: actions/cache@v4 + with: +path: ${{ inputs.maven_repo_folder }} +key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} +restore-keys: ${{ runner.os }}-maven + +- name: "Moves checkout content from ${{ inputs.source_directory }} to ${{ inputs.target_directory }}." + if: ${{ inputs.source_directory != '' && inputs.target_directory != '' }} + shell: bash + run: | +mkdir -p ${{ inputs.target_directory }} + +# .scalafmt.conf is needed for Scala format checks +# .mvn is needed to make the Maven wrapper accessible in test runs +mv ${{ inputs.source_directory }}/* \ + ${{ inputs.source_directory }}/.scalafmt.conf \ + ${{ inputs.source_directory }}/.mvn \ + ${{ inputs.target_directory }} diff --git a/.github/actions/run_mvn/action.yml b/.github/actions/run_mvn/action.yml new file mode 100644 index 000..03595a25cdb --- /dev/null +++ b/.github/actions/run_mvn/action.yml @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribu
(flink) 02/08: [hotfix][ci] Fixes condition for filtering NOTICE files
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit f10702b8c235a3e4fb25f7efc89da5d775e74b2d Author: Matthias Pohl AuthorDate: Fri Dec 8 17:17:05 2023 +0100 [hotfix][ci] Fixes condition for filtering NOTICE files --- .../java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java index 9a8b10db810..e39bcda1c50 100644 --- a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java +++ b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java @@ -359,7 +359,8 @@ public class NoticeFileChecker { .filter( file -> { int nameCount = file.getNameCount(); -return file.getName(nameCount - 3).toString().equals("resources") +return nameCount >= 3 +&& file.getName(nameCount - 3).toString().equals("resources") && file.getName(nameCount - 2).toString().equals("META-INF") && file.getName(nameCount - 1).toString().equals("NOTICE"); })
(flink) 05/08: [hotfix][ci] Adds proper log message at the end of a successful CI run
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit e4b4718009a2237fcb4b77df3573c9c96d747ce7 Author: Matthias Pohl AuthorDate: Wed Jan 24 10:31:30 2024 +0100 [hotfix][ci] Adds proper log message at the end of a successful CI run --- tools/azure-pipelines/uploading_watchdog.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/azure-pipelines/uploading_watchdog.sh b/tools/azure-pipelines/uploading_watchdog.sh index 2eafc1dfcbe..dc3d227ecbe 100755 --- a/tools/azure-pipelines/uploading_watchdog.sh +++ b/tools/azure-pipelines/uploading_watchdog.sh @@ -105,8 +105,8 @@ WATCHDOG_PID=$! ( $COMMAND & PID=$! ; echo $PID >$MAIN_PID_FILE ; wait $PID ) | ts | tee $DEBUG_FILES_OUTPUT_DIR/watchdog TEST_EXIT_CODE=${PIPESTATUS[0]} -# successful execution, cleanup watchdog related things if [[ "$TEST_EXIT_CODE" == 0 ]]; then + echo "[INFO] Test execution finished with exit code 0. Artifacts related to the watchdog process will be cleaned up." kill $WATCHDOG_PID rm $DEBUG_FILES_OUTPUT_DIR/watchdog rm -f $DEBUG_FILES_OUTPUT_DIR/jps-traces.*
(flink) 07/08: [FLINK-27082][ci] Updates github-actions profile.
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit a810088688c7fe1bc969ed99aed3aaaf73742449 Author: Matthias Pohl AuthorDate: Fri Jan 5 15:48:03 2024 +0100 [FLINK-27082][ci] Updates github-actions profile. A dedicated github-actions profile doesn't work if other profiles which configure excludeGroups for the surefire plugin are enabled. --- pom.xml | 46 ++ 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/pom.xml b/pom.xml index f7de4a97b5b..9c20d806d77 100644 --- a/pom.xml +++ b/pom.xml @@ -204,6 +204,10 @@ under the License. of entries in the form '[-]{2}add-[opens|exports]=/=ALL-UNNAMED'.--> + + + + **/*Test.* @@ -1000,6 +1004,9 @@ under the License. enable-adaptive-scheduler + + org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler + @@ -1009,7 +1016,11 @@ under the License. true - org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk, + @@ -1018,7 +1029,9 @@ under the License. github-actions - + + org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser + @@ -1026,7 +1039,11 @@ under the License. org.apache.maven.plugins maven-surefire-plugin - org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk}, + @@ -1040,6 +1057,10 @@ under the License. [11,) + + org.apache.flink.testutils.junit.FailsOnJava11 + + @@ -1052,7 +1073,11 @@ under the License. org.apache.maven.plugins maven-surefire-plugin - org.apache.flink.testutils.junit.FailsOnJava11 + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk}, + @@ -1101,6 +1126,7 @@ under the License.
(flink) 06/08: [FLINK-34062][build] Adds JDK 21 surefire configuration.
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit f4c0b716b01200d5e02bb49d9d859934867080a4 Author: Matthias Pohl AuthorDate: Thu Jan 11 12:50:38 2024 +0100 [FLINK-34062][build] Adds JDK 21 surefire configuration. --- pom.xml | 11 +++ 1 file changed, 11 insertions(+) diff --git a/pom.xml b/pom.xml index fbf34fcd46b..f7de4a97b5b 100644 --- a/pom.xml +++ b/pom.xml @@ -1169,6 +1169,17 @@ under the License. true + + org.apache.maven.plugins + maven-surefire-plugin + + + ${github-actions.surefire.excludedGroups}, + org.apache.flink.testutils.junit.FailsOnJava11, + org.apache.flink.testutils.junit.FailsOnJava17, + + +
(flink) 04/08: [hotfix][ci] Replaces spaces with tabs
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit edfdc89a029fce0fa87fcec08eaefb0fc3745c9f Author: Matthias Pohl AuthorDate: Wed Jan 24 10:12:19 2024 +0100 [hotfix][ci] Replaces spaces with tabs The space indentation was accidentally added in FLINK-33906 --- tools/azure-pipelines/debug_files_utils.sh | 44 +++--- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/tools/azure-pipelines/debug_files_utils.sh b/tools/azure-pipelines/debug_files_utils.sh index 6b74912e9fe..c2b3c3d5e8e 100755 --- a/tools/azure-pipelines/debug_files_utils.sh +++ b/tools/azure-pipelines/debug_files_utils.sh @@ -30,28 +30,28 @@ function prepare_debug_files { export DEBUG_FILES_OUTPUT_DIR="${parent_directory}/debug_files" export DEBUG_FILES_NAME="$(echo "${module}" | tr -c '[:alnum:]\n\r' '_')-$(date +%s)" - if [ -n "${TF_BUILD+x}" ]; then -echo "[INFO] Azure Pipelines environment detected: $0 will export the variables in the Azure-specific way." - -echo "##vso[task.setvariable variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR" -echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME" - elif [ -n "${GITHUB_ACTIONS+x}" ]; then -echo "[INFO] GitHub Actions environment detected: $0 will export the variables in the GHA-specific way." - -if [ -z "${GITHUB_OUTPUT+x}" ]; then - echo "[ERROR] The GITHUB_OUTPUT variable is not set." - exit 1 -elif [ ! -f "$GITHUB_OUTPUT" ]; then - echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file: $GITHUB_OUTPUT" - exit 1 -fi - -echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT" -echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT" - else -echo "[ERROR] No CI environment detected. Debug artifact-related variables couldn't be exported." -exit 1 - fi + if [ -n "${TF_BUILD+x}" ]; then + echo "[INFO] Azure Pipelines environment detected: $0 will export the variables in the Azure-specific way." + + echo "##vso[task.setvariable variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR" + echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME" + elif [ -n "${GITHUB_ACTIONS+x}" ]; then + echo "[INFO] GitHub Actions environment detected: $0 will export the variables in the GHA-specific way." + + if [ -z "${GITHUB_OUTPUT+x}" ]; then + echo "[ERROR] The GITHUB_OUTPUT variable is not set." + exit 1 + elif [ ! -f "$GITHUB_OUTPUT" ]; then + echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file: $GITHUB_OUTPUT" + exit 1 + fi + + echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT" + echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT" + else + echo "[ERROR] No CI environment detected. Debug artifact-related variables couldn't be exported." + exit 1 + fi mkdir -p $DEBUG_FILES_OUTPUT_DIR || { echo "FAILURE: cannot create debug files directory '${DEBUG_FILES_OUTPUT_DIR}'." ; exit 1; } }
(flink) 06/07: [FLINK-27082][ci] Updates github-actions profile.
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit fff63f4443e2f98ccd49f6cdbb46a28e52371d38 Author: Matthias Pohl AuthorDate: Fri Jan 5 15:48:03 2024 +0100 [FLINK-27082][ci] Updates github-actions profile. A dedicated github-actions profile doesn't work if other profiles which configure excludeGroups for the surefire plugin are enabled. --- pom.xml | 40 +++- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/pom.xml b/pom.xml index 8547171216a..3246cf9a81a 100644 --- a/pom.xml +++ b/pom.xml @@ -202,6 +202,10 @@ under the License. of entries in the form '[-]{2}add-[opens|exports]=/=ALL-UNNAMED'.--> + + + + **/*Test.* @@ -1000,6 +1004,9 @@ under the License. enable-adaptive-scheduler + + org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler + @@ -1009,7 +1016,11 @@ under the License. true - org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk, + @@ -1018,7 +1029,9 @@ under the License. github-actions - + + org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser + @@ -1026,7 +1039,11 @@ under the License. org.apache.maven.plugins maven-surefire-plugin - org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk}, + @@ -1040,6 +1057,10 @@ under the License. [11,) + + org.apache.flink.testutils.junit.FailsOnJava11 + + @@ -1052,7 +1073,11 @@ under the License. org.apache.maven.plugins maven-surefire-plugin - org.apache.flink.testutils.junit.FailsOnJava11 + + ${surefire.excludedGroups.github-actions}, + ${surefire.excludedGroups.adaptive-scheduler}, + ${surefire.excludedGroups.jdk}, + @@ -1101,6 +1126,7 @@ under the License.
(flink) branch release-1.18 updated (d903990e1ed -> a64e1c98533)
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a change to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git from d903990e1ed [FLINK-34115][table-planner] Fix TableAggregateITCase unstable test new 2c6493bcc1e [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found new 227f50b2b66 [hotfix][ci] Fixes condition for filtering NOTICE files new ffdce2cef6a [hotfix][build] Adds Maven and Java version to Maven command new bfa590d1b52 [hotfix][ci] Replaces spaces with tabs new 4da0f1b05eb [hotfix][ci] Adds proper log message at the end of a successful CI run new fff63f4443e [FLINK-27082][ci] Updates github-actions profile. new a64e1c98533 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/actions/job_init/action.yml| 72 .github/actions/run_mvn/action.yml | 42 +++ .../workflows/ci.yml | 57 ++- .github/workflows/template.flink-ci.yml| 386 + .github/workflows/template.pre-compile-checks.yml | 77 pom.xml| 40 ++- tools/azure-pipelines/create_build_artifact.sh | 12 +- tools/azure-pipelines/debug_files_utils.sh | 62 ++-- tools/azure-pipelines/unpack_build_artifact.sh | 20 +- tools/azure-pipelines/uploading_watchdog.sh| 4 +- .../tools/ci/licensecheck/NoticeFileChecker.java | 3 +- tools/ci/maven-utils.sh| 1 + 12 files changed, 705 insertions(+), 71 deletions(-) create mode 100644 .github/actions/job_init/action.yml create mode 100644 .github/actions/run_mvn/action.yml copy tools/releasing/deploy_staging_jars.sh => .github/workflows/ci.yml (52%) mode change 100755 => 100644 create mode 100644 .github/workflows/template.flink-ci.yml create mode 100644 .github/workflows/template.pre-compile-checks.yml
(flink) 01/07: [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit 2c6493bcc1ef643330b59bea77bc5f3d703f9d85 Author: Matthias Pohl AuthorDate: Thu Dec 21 18:27:46 2023 +0100 [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found --- tools/azure-pipelines/create_build_artifact.sh | 2 +- tools/azure-pipelines/unpack_build_artifact.sh | 11 +-- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/tools/azure-pipelines/create_build_artifact.sh b/tools/azure-pipelines/create_build_artifact.sh index 809d14fe651..0beeca3b075 100755 --- a/tools/azure-pipelines/create_build_artifact.sh +++ b/tools/azure-pipelines/create_build_artifact.sh @@ -28,7 +28,7 @@ echo "Minimizing artifact files" # by removing files not required for subsequent stages # jars are re-built in subsequent stages, so no need to cache them (cannot be avoided) -find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' | xargs rm -rf +find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' -exec rm -rf {} \; # .git directory # not deleting this can cause build stability issues diff --git a/tools/azure-pipelines/unpack_build_artifact.sh b/tools/azure-pipelines/unpack_build_artifact.sh index 15d5c7c1057..68f8d221aae 100755 --- a/tools/azure-pipelines/unpack_build_artifact.sh +++ b/tools/azure-pipelines/unpack_build_artifact.sh @@ -28,15 +28,14 @@ cp -RT "$FLINK_ARTIFACT_DIR" "." echo "Adjusting timestamps" # adjust timestamps of proto file to avoid re-generation -find . -type f -name '*.proto' | xargs touch +find . -type f -name '*.proto' -exec touch {} \; # wait a bit for better odds of different timestamps sleep 5 # adjust timestamps to prevent recompilation -find . -type f -name '*.java' | xargs touch -find . -type f -name '*.scala' | xargs touch +find . -type f -name '*.java' -exec touch {} \; +find . -type f -name '*.scala' -exec touch {} \; # wait a bit for better odds of different timestamps sleep 5 -find . -type f -name '*.class' | xargs touch -find . -type f -name '*.timestamp' | xargs touch - +find . -type f -name '*.class' -exec touch {} \; +find . -type f -name '*.timestamp' -exec touch {} \;
(flink) 04/07: [hotfix][ci] Replaces spaces with tabs
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit bfa590d1b52fd70baf1918821f5cd8bd219e8785 Author: Matthias Pohl AuthorDate: Wed Jan 24 10:12:19 2024 +0100 [hotfix][ci] Replaces spaces with tabs The space indentation was accidentally added in FLINK-33906 --- tools/azure-pipelines/debug_files_utils.sh | 44 +++--- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/tools/azure-pipelines/debug_files_utils.sh b/tools/azure-pipelines/debug_files_utils.sh index 6b74912e9fe..c2b3c3d5e8e 100755 --- a/tools/azure-pipelines/debug_files_utils.sh +++ b/tools/azure-pipelines/debug_files_utils.sh @@ -30,28 +30,28 @@ function prepare_debug_files { export DEBUG_FILES_OUTPUT_DIR="${parent_directory}/debug_files" export DEBUG_FILES_NAME="$(echo "${module}" | tr -c '[:alnum:]\n\r' '_')-$(date +%s)" - if [ -n "${TF_BUILD+x}" ]; then -echo "[INFO] Azure Pipelines environment detected: $0 will export the variables in the Azure-specific way." - -echo "##vso[task.setvariable variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR" -echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME" - elif [ -n "${GITHUB_ACTIONS+x}" ]; then -echo "[INFO] GitHub Actions environment detected: $0 will export the variables in the GHA-specific way." - -if [ -z "${GITHUB_OUTPUT+x}" ]; then - echo "[ERROR] The GITHUB_OUTPUT variable is not set." - exit 1 -elif [ ! -f "$GITHUB_OUTPUT" ]; then - echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file: $GITHUB_OUTPUT" - exit 1 -fi - -echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT" -echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT" - else -echo "[ERROR] No CI environment detected. Debug artifact-related variables couldn't be exported." -exit 1 - fi + if [ -n "${TF_BUILD+x}" ]; then + echo "[INFO] Azure Pipelines environment detected: $0 will export the variables in the Azure-specific way." + + echo "##vso[task.setvariable variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR" + echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME" + elif [ -n "${GITHUB_ACTIONS+x}" ]; then + echo "[INFO] GitHub Actions environment detected: $0 will export the variables in the GHA-specific way." + + if [ -z "${GITHUB_OUTPUT+x}" ]; then + echo "[ERROR] The GITHUB_OUTPUT variable is not set." + exit 1 + elif [ ! -f "$GITHUB_OUTPUT" ]; then + echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file: $GITHUB_OUTPUT" + exit 1 + fi + + echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT" + echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT" + else + echo "[ERROR] No CI environment detected. Debug artifact-related variables couldn't be exported." + exit 1 + fi mkdir -p $DEBUG_FILES_OUTPUT_DIR || { echo "FAILURE: cannot create debug files directory '${DEBUG_FILES_OUTPUT_DIR}'." ; exit 1; } }
(flink) 05/07: [hotfix][ci] Adds proper log message at the end of a successful CI run
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit 4da0f1b05ebd192db58b41886d396fa68eca6e01 Author: Matthias Pohl AuthorDate: Wed Jan 24 10:31:30 2024 +0100 [hotfix][ci] Adds proper log message at the end of a successful CI run --- tools/azure-pipelines/uploading_watchdog.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/azure-pipelines/uploading_watchdog.sh b/tools/azure-pipelines/uploading_watchdog.sh index 2eafc1dfcbe..dc3d227ecbe 100755 --- a/tools/azure-pipelines/uploading_watchdog.sh +++ b/tools/azure-pipelines/uploading_watchdog.sh @@ -105,8 +105,8 @@ WATCHDOG_PID=$! ( $COMMAND & PID=$! ; echo $PID >$MAIN_PID_FILE ; wait $PID ) | ts | tee $DEBUG_FILES_OUTPUT_DIR/watchdog TEST_EXIT_CODE=${PIPESTATUS[0]} -# successful execution, cleanup watchdog related things if [[ "$TEST_EXIT_CODE" == 0 ]]; then + echo "[INFO] Test execution finished with exit code 0. Artifacts related to the watchdog process will be cleaned up." kill $WATCHDOG_PID rm $DEBUG_FILES_OUTPUT_DIR/watchdog rm -f $DEBUG_FILES_OUTPUT_DIR/jps-traces.*
(flink) 02/07: [hotfix][ci] Fixes condition for filtering NOTICE files
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit 227f50b2b66ec8963d637d8929088aca91d88c04 Author: Matthias Pohl AuthorDate: Fri Dec 8 17:17:05 2023 +0100 [hotfix][ci] Fixes condition for filtering NOTICE files --- .../java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java index 9a8b10db810..e39bcda1c50 100644 --- a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java +++ b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java @@ -359,7 +359,8 @@ public class NoticeFileChecker { .filter( file -> { int nameCount = file.getNameCount(); -return file.getName(nameCount - 3).toString().equals("resources") +return nameCount >= 3 +&& file.getName(nameCount - 3).toString().equals("resources") && file.getName(nameCount - 2).toString().equals("META-INF") && file.getName(nameCount - 1).toString().equals("NOTICE"); })
(flink) 07/07: [FLINK-33914][ci] Introduces a basic GitHub Actions workflow
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit a64e1c98533649d8567f416fb94432d93ddf1d28 Author: Matthias Pohl AuthorDate: Sun Dec 10 13:36:57 2023 +0100 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow - Adds basic CI workflow - Adds composite workflow for starting Flink's CI stages - Updates CI helper scripts to support GitHub Actions - Adds composite workflow for doing pre-compile checks - Adds custom action to enable maven-utils.sh in workflows - Adds custom action for initializing a job --- .github/actions/job_init/action.yml | 72 .github/actions/run_mvn/action.yml| 42 +++ .github/workflows/ci.yml | 40 +++ .github/workflows/template.flink-ci.yml | 386 ++ .github/workflows/template.pre-compile-checks.yml | 77 + tools/azure-pipelines/create_build_artifact.sh| 10 +- tools/azure-pipelines/debug_files_utils.sh| 18 + tools/azure-pipelines/unpack_build_artifact.sh| 9 +- tools/azure-pipelines/uploading_watchdog.sh | 2 + 9 files changed, 653 insertions(+), 3 deletions(-) diff --git a/.github/actions/job_init/action.yml b/.github/actions/job_init/action.yml new file mode 100644 index 000..5158d855857 --- /dev/null +++ b/.github/actions/job_init/action.yml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Initializes the job" +description: "Does all the necessary steps to set up the job" +inputs: + jdk_version: +description: "The JDK version that's supposed to be used." +required: true + maven_repo_folder: +description: "The location of the local Maven repository (not setting this parameter will omit the caching of Maven artifacts)." +required: false +default: "" + source_directory: +description: "Specifies the directory from which the code should be moved from (needed for containerized runs; not setting this parameter will omit moving the checkout)." +required: false +default: "" + target_directory: +description: "Specifies the directory to which the code should be moved to (needed for containerized runs; not setting this parameter will omit moving the checkout)." +required: false +default: "" +runs: + using: "composite" + steps: +- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time" + shell: bash + run: | +job_start_time="$(date --rfc-3339=seconds)" +echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}" +echo "The job's start time is set to ${job_start_time}." + +- name: "Set JDK version to ${{ inputs.jdk_version }}" + shell: bash + run: | +echo "JAVA_HOME=$JAVA_HOME_${{ inputs.jdk_version }}_X64" >> "${GITHUB_ENV}" +echo "PATH=$JAVA_HOME_${{ inputs.jdk_version }}_X64/bin:$PATH" >> "${GITHUB_ENV}" + +- name: "Setup Maven package cache" + if: ${{ inputs.maven_repo_folder != '' }} + uses: actions/cache@v4 + with: +path: ${{ inputs.maven_repo_folder }} +key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} +restore-keys: ${{ runner.os }}-maven + +- name: "Moves checkout content from ${{ inputs.source_directory }} to ${{ inputs.target_directory }}." + if: ${{ inputs.source_directory != '' && inputs.target_directory != '' }} + shell: bash + run: | +mkdir -p ${{ inputs.target_directory }} + +# .scalafmt.conf is needed for Scala format checks +# .mvn is needed to make the Maven wrapper accessible in test runs +mv ${{ inputs.source_directory }}/* \ + ${{ inputs.source_directory }}/.scalafmt.conf \ + ${{ inputs.source_directory }}/.mvn \ + ${{ inputs.target_directory }} diff --git a/.github/actions/run_mvn/action.yml b/.github/actions/run_mvn/action.yml new file mode 100644 index 000..03595a25cdb --- /dev/null +++ b/.github/actions/run_mvn/action.yml @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co
(flink) 03/07: [hotfix][build] Adds Maven and Java version to Maven command
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git commit ffdce2cef6a56db77be209d5721c2f6fd3208eea Author: Matthias Pohl AuthorDate: Tue Jan 16 16:20:45 2024 +0100 [hotfix][build] Adds Maven and Java version to Maven command --- tools/ci/maven-utils.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/ci/maven-utils.sh b/tools/ci/maven-utils.sh index 2ac16179b23..ab535af4320 100755 --- a/tools/ci/maven-utils.sh +++ b/tools/ci/maven-utils.sh @@ -17,6 +17,7 @@ # Utility for invoking Maven in CI function run_mvn { if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then + $MAVEN_WRAPPER --version echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'" fi $MAVEN_WRAPPER $MVN_GLOBAL_OPTIONS "${@}"
(flink) branch master updated: [FLINK-33915][ci] Adds workflow for extended master CI workflow
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 697d328f6bb [FLINK-33915][ci] Adds workflow for extended master CI workflow 697d328f6bb is described below commit 697d328f6bb090db63b80f063d3298446a33a5b9 Author: Matthias Pohl AuthorDate: Fri Dec 15 10:46:13 2023 +0100 [FLINK-33915][ci] Adds workflow for extended master CI workflow --- .github/workflows/nightly.yml | 98 +++ 1 file changed, 98 insertions(+) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 000..4f2603a6c27 --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,98 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This workflow is meant as an extended CI run that includes certain features that shall be tested +# and JDK versions that are supported but not considered default. + +name: "Nightly (beta)" + +on: + schedule: +- cron: '0 2 * * *' + workflow_dispatch: + +permissions: read-all + +jobs: + pre-compile-checks: +name: "Pre-compile Checks" +uses: ./.github/workflows/template.pre-compile-checks.yml + + java8: +name: "Java 8" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java8 + environment: 'PROFILE="-Dinclude_hadoop_aws"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + java11: +name: "Java 11" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java11 + environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"' + jdk_version: 11 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + java17: +name: "Java 17" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java17 + environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Pjava17-target"' + jdk_version: 17 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + java21: +name: "Java 21" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java21 + environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21 -Pjava21-target"' + jdk_version: 21 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + hadoop313: +name: "Hadoop 3.1.3" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: hadoop313 + environment: 'PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + adaptive-scheduler: +name: "AdaptiveScheduler" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: adaptive-scheduler + environment: 'PROFILE="-Penable-adaptive-scheduler"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
(flink) branch release-1.18 updated: [FLINK-33915][ci] Adds workflow release-1.18 nightly CI workflow
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a commit to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.18 by this push: new a27e03650d9 [FLINK-33915][ci] Adds workflow release-1.18 nightly CI workflow a27e03650d9 is described below commit a27e03650d9e03c9317e19051f8aa4871f199f4a Author: Matthias Pohl AuthorDate: Fri Dec 15 10:46:13 2023 +0100 [FLINK-33915][ci] Adds workflow release-1.18 nightly CI workflow --- .github/workflows/nightly.yml | 85 +++ 1 file changed, 85 insertions(+) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 000..c02da2e4954 --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,85 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This workflow is meant as an extended CI run that includes certain features that shall be tested +# and JDK versions that are supported but not considered default. + +name: "Nightly (beta)" + +on: + workflow_dispatch: + +permissions: read-all + +jobs: + pre-compile-checks: +name: "Pre-compile Checks" +uses: ./.github/workflows/template.pre-compile-checks.yml + + java8: +name: "Java 8" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java8 + environment: 'PROFILE="-Dinclude_hadoop_aws"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + java11: +name: "Java 11" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java11 + environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"' + jdk_version: 11 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + java17: +name: "Java 17" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: java17 + environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Pjava17-target"' + jdk_version: 17 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + hadoop313: +name: "Hadoop 3.1.3" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: hadoop313 + environment: 'PROFILE="-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }} + adaptive-scheduler: +name: "AdaptiveScheduler" +uses: ./.github/workflows/template.flink-ci.yml +with: + workflow-caller-id: adaptive-scheduler + environment: 'PROFILE="-Penable-adaptive-scheduler"' + jdk_version: 8 +secrets: + s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }} + s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }} + s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
(flink) branch master updated (697d328f6bb -> ff9a3c85f69)
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 697d328f6bb [FLINK-33915][ci] Adds workflow for extended master CI workflow add ff9a3c85f69 [FLINK-33916][ci] Adds workflow trigger No new revisions were added by this update. Summary of changes: .github/workflows/{ci.yml => nightly-trigger.yml} | 45 +-- .github/workflows/nightly.yml | 2 - 2 files changed, 25 insertions(+), 22 deletions(-) copy .github/workflows/{ci.yml => nightly-trigger.yml} (51%)
(flink) branch master updated: [FLINK-34316] Reduce instantiation of ScanRuntimeProvider in streaming mode
This is an automated email from the ASF dual-hosted git repository. twalthr pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 503580834e0 [FLINK-34316] Reduce instantiation of ScanRuntimeProvider in streaming mode 503580834e0 is described below commit 503580834e06455bbcaf619a51f8d18812bbe83f Author: Timo Walther AuthorDate: Wed Jan 31 09:34:48 2024 +0100 [FLINK-34316] Reduce instantiation of ScanRuntimeProvider in streaming mode --- .../apache/flink/table/planner/connectors/DynamicSourceUtils.java | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java index 925d1a00b11..2ac8aeafe75 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java @@ -498,14 +498,12 @@ public final class DynamicSourceUtils { ScanTableSource scanSource, boolean isBatchMode, ReadableConfig config) { -final ScanRuntimeProvider provider = - scanSource.getScanRuntimeProvider(ScanRuntimeProviderContext.INSTANCE); final ChangelogMode changelogMode = scanSource.getChangelogMode(); validateWatermarks(tableDebugName, schema); if (isBatchMode) { -validateScanSourceForBatch(tableDebugName, changelogMode, provider); +validateScanSourceForBatch(tableDebugName, scanSource, changelogMode); } else { validateScanSourceForStreaming( tableDebugName, schema, scanSource, changelogMode, config); @@ -558,7 +556,9 @@ public final class DynamicSourceUtils { } private static void validateScanSourceForBatch( -String tableDebugName, ChangelogMode changelogMode, ScanRuntimeProvider provider) { +String tableDebugName, ScanTableSource scanSource, ChangelogMode changelogMode) { +final ScanRuntimeProvider provider = + scanSource.getScanRuntimeProvider(ScanRuntimeProviderContext.INSTANCE); // batch only supports bounded source if (!provider.isBounded()) { throw new ValidationException(
(flink) branch master updated: [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT
This is an automated email from the ASF dual-hosted git repository. jchan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new cb9e220c229 [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT cb9e220c229 is described below commit cb9e220c2291088459f0281aa8e8e8584436a9b2 Author: Xuyang AuthorDate: Wed Jan 31 21:32:43 2024 +0800 [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT This closes #24226 --- .../exec/stream/GroupAggregateTestPrograms.java | 10 ++ .../plan/agg-with-state-ttl-hint.json | 2 +- .../agg-with-state-ttl-hint/savepoint/_metadata | Bin 11151 -> 11151 bytes 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java index 6c43cf90959..755eb781a35 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java @@ -387,13 +387,15 @@ public class GroupAggregateTestPrograms { "+I[1, 1, null, Hi]", "+I[2, 1, 2.0, Hello]", "+U[2, 2, 2.0, Hello]") +// Due to state TTL in hint, the state in the metadata +// savepoint has expired. .consumedAfterRestore( -"+U[1, 2, null, Hi]", -"+U[2, 3, 2.0, Hello]", -"+U[2, 4, 2.0, Hello]") +"+I[1, 1, null, Hi Again!]", +"+I[2, 1, 2.0, Hello Again!]", +"+U[2, 2, 2.0, Hello Again!]") .build()) .runSql( -"INSERT INTO sink_t SELECT /*+ STATE_TTL('source_t' = '4d') */" +"INSERT INTO sink_t SELECT /*+ STATE_TTL('source_t' = '1s') */" + "b, " + "COUNT(*) AS cnt, " + "AVG(a) FILTER (WHERE a > 1) AS avg_a, " diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json index cebe1e5898b..cfdcfed459b 100644 --- a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json +++ b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json @@ -128,7 +128,7 @@ "needRetraction" : false, "state" : [ { "index" : 0, - "ttl" : "4 d", + "ttl" : "1 s", "name" : "groupAggregateState" } ], "inputProperties" : [ { diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata index 50e4bee3a94..e956dafb981 100644 Binary files a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata and b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata differ
(flink-kubernetes-operator) branch main updated: [FLINK-34311] Do not change min resource requirements when rescaling for adaptive scheduler
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git The following commit(s) were added to refs/heads/main by this push: new 4342636c [FLINK-34311] Do not change min resource requirements when rescaling for adaptive scheduler 4342636c is described below commit 4342636cdb2c3439389e83cb4fe4366156edfbd7 Author: Gyula Fora AuthorDate: Tue Jan 30 15:43:46 2024 +0100 [FLINK-34311] Do not change min resource requirements when rescaling for adaptive scheduler --- .../kubernetes/operator/service/NativeFlinkService.java | 12 +--- .../kubernetes/operator/service/NativeFlinkServiceTest.java | 10 +++--- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java index 9781a5b0..41083d91 100644 --- a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java +++ b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java @@ -200,9 +200,15 @@ public class NativeFlinkService extends AbstractFlinkService { var overrideStr = newOverrides.get(jobId); if (overrideStr != null) { -// We have an override for the vertex -int p = Integer.parseInt(overrideStr); -var newParallelism = new JobVertexResourceRequirements.Parallelism(p, p); +// We set the parallelism upper bound to the target parallelism, anything higher +// would defeat the purpose of scaling down +int upperBound = Integer.parseInt(overrideStr); +// We only change the lower bound if the new parallelism went below it. As we +// cannot guarantee that new resources can be acquired, increasing the lower +// bound to the target could potentially cause job failure. +int lowerBound = Math.min(upperBound, parallelism.getLowerBound()); +var newParallelism = +new JobVertexResourceRequirements.Parallelism(lowerBound, upperBound); // If the requirements changed we mark this as scaling triggered if (!parallelism.equals(newParallelism)) { entry.setValue(new JobVertexResourceRequirements(newParallelism)); diff --git a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java index c8c0afb3..693a9019 100644 --- a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java +++ b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java @@ -229,7 +229,9 @@ public class NativeFlinkServiceTest { var reconStatus = flinkDep.getStatus().getReconciliationStatus(); reconStatus.serializeAndSetLastReconciledSpec(spec, flinkDep); -appConfig.set(PipelineOptions.PARALLELISM_OVERRIDES, Map.of(v1.toHexString(), "4")); +appConfig.set( +PipelineOptions.PARALLELISM_OVERRIDES, +Map.of(v1.toHexString(), "4", v2.toHexString(), "1")); spec.setFlinkConfiguration(appConfig.toMap()); flinkDep.getStatus().getJobStatus().setState("RUNNING"); @@ -256,13 +258,15 @@ public class NativeFlinkServiceTest { Map.of( v1, new JobVertexResourceRequirements( -new JobVertexResourceRequirements.Parallelism(4, 4)), +new JobVertexResourceRequirements.Parallelism(1, 4)), v2, new JobVertexResourceRequirements( -new JobVertexResourceRequirements.Parallelism(2, 2))), +new JobVertexResourceRequirements.Parallelism(1, 1))), updated.get()); // Baseline +appConfig.set(PipelineOptions.PARALLELISM_OVERRIDES, Map.of(v1.toHexString(), "4")); +spec.setFlinkConfiguration(appConfig.toMap()); testScaleConditionDep( flinkDep, service, d -> {}, FlinkService.ScalingResult.SCALING_TRIGGERED); testScaleConditionLastSpec(
(flink-connector-jdbc) branch v3.1 updated: [hotfix] Use 1.16.x as minimum supported version in pom
This is an automated email from the ASF dual-hosted git repository. snuyanzin pushed a commit to branch v3.1 in repository https://gitbox.apache.org/repos/asf/flink-connector-jdbc.git The following commit(s) were added to refs/heads/v3.1 by this push: new 390d7bc9 [hotfix] Use 1.16.x as minimum supported version in pom 390d7bc9 is described below commit 390d7bc9139204fbfc48fe275a69eb60c4807fb5 Author: Sergey Nuyanzin AuthorDate: Wed Jan 31 16:44:11 2024 +0100 [hotfix] Use 1.16.x as minimum supported version in pom --- .github/workflows/push_pr.yml | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml index a54f4a0d..38724aef 100644 --- a/.github/workflows/push_pr.yml +++ b/.github/workflows/push_pr.yml @@ -25,7 +25,7 @@ jobs: compile_and_test: strategy: matrix: -flink: [1.17-SNAPSHOT] +flink: [ 1.16-SNAPSHOT, 1.17-SNAPSHOT ] jdk: [ '8, 11' ] include: - flink: 1.18-SNAPSHOT diff --git a/pom.xml b/pom.xml index dd645500..9d44d460 100644 --- a/pom.xml +++ b/pom.xml @@ -47,7 +47,7 @@ under the License. -1.17.0 +1.16.0 2.13.4.20221013 4.13.2
(flink) branch master updated (cb9e220c229 -> 8e11b6005c9)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from cb9e220c229 [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT add 8e11b6005c9 [FLINK-34272][runtime] ExecutingState should never transition during its construction. No new revisions were added by this update. Summary of changes: .../runtime/scheduler/adaptive/Executing.java | 2 +- .../runtime/scheduler/adaptive/ExecutingTest.java | 109 + .../MockStateWithExecutionGraphContext.java| 5 +- 3 files changed, 50 insertions(+), 66 deletions(-)
(flink) branch master updated (8e11b6005c9 -> 98e1d079f6e)
This is an automated email from the ASF dual-hosted git repository. libenchao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 8e11b6005c9 [FLINK-34272][runtime] ExecutingState should never transition during its construction. add 3c8088b716a [hotfix][table-common] Mark SinkV2Provider as the recommended core interface for SinkRuntimeProvider add 98e1d079f6e [FLINK-34270][docs] Update doc for the new Table source & sink interfaces introduced in FLIP-146/367 No new revisions were added by this update. Summary of changes: docs/content.zh/docs/dev/table/sourcesSinks.md | 12 ++-- docs/content/docs/dev/table/sourcesSinks.md| 22 ++ .../table/connector/sink/DynamicTableSink.java | 16 3 files changed, 36 insertions(+), 14 deletions(-)
(flink) branch master updated: [FLINK-33358][sql-client] Fix Flink SQL Client fail to start in Flink on YARN (#23629)
This is an automated email from the ASF dual-hosted git repository. shengkai pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new fa364c7c668 [FLINK-33358][sql-client] Fix Flink SQL Client fail to start in Flink on YARN (#23629) fa364c7c668 is described below commit fa364c7c668ec2a87bcbf18ce2b80f749cc16b2b Author: Prabhu Joseph AuthorDate: Thu Feb 1 09:21:40 2024 +0530 [FLINK-33358][sql-client] Fix Flink SQL Client fail to start in Flink on YARN (#23629) Co-authored-by: Prabhu Joseph --- flink-table/flink-sql-client/bin/sql-client.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flink-table/flink-sql-client/bin/sql-client.sh b/flink-table/flink-sql-client/bin/sql-client.sh index 3091bdf1b36..797dcc4ac05 100755 --- a/flink-table/flink-sql-client/bin/sql-client.sh +++ b/flink-table/flink-sql-client/bin/sql-client.sh @@ -87,7 +87,7 @@ if [[ "$CC_CLASSPATH" =~ .*flink-sql-client.*.jar ]]; then elif [ -n "$FLINK_SQL_CLIENT_JAR" ]; then # start client with jar -exec "$JAVA_RUN" $FLINK_ENV_JAVA_OPTS $JVM_ARGS "${log_setting[@]}" -classpath "`manglePathList "$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS:$FLINK_SQL_CLIENT_JAR"`" org.apache.flink.table.client.SqlClient "$@" --jar "`manglePath $FLINK_SQL_CLIENT_JAR`" +exec "$JAVA_RUN" $FLINK_ENV_JAVA_OPTS $JVM_ARGS "${log_setting[@]}" -classpath "`manglePathList "$CC_CLASSPATH:$FLINK_SQL_CLIENT_JAR:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.table.client.SqlClient "$@" --jar "`manglePath $FLINK_SQL_CLIENT_JAR`" # write error message to stderr else
(flink) branch master updated: [FLINK-34206][runtime] Fix potential job failure due to concurrent global failure and source paralelism inference
This is an automated email from the ASF dual-hosted git repository. zhuzh pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new b737b718596 [FLINK-34206][runtime] Fix potential job failure due to concurrent global failure and source paralelism inference b737b718596 is described below commit b737b71859672e8020881ce2abf998735ee98abb Author: sunxia AuthorDate: Tue Jan 30 14:26:26 2024 +0800 [FLINK-34206][runtime] Fix potential job failure due to concurrent global failure and source paralelism inference This closes #24223. --- .../runtime/scheduler/ExecutionVertexVersioner.java | 2 +- .../scheduler/adaptivebatch/AdaptiveBatchScheduler.java | 16 ++-- .../apache/flink/test/streaming/runtime/CacheITCase.java | 2 -- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java index b0a0b17db0d..a86d22fc40c 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java @@ -92,7 +92,7 @@ public class ExecutionVertexVersioner { ExecutionVertexVersion::getExecutionVertexId, Function.identity())); } -ExecutionVertexVersion getExecutionVertexVersion(ExecutionVertexID executionVertexId) { +public ExecutionVertexVersion getExecutionVertexVersion(ExecutionVertexID executionVertexId) { final long currentVersion = getCurrentVersion(executionVertexId); return new ExecutionVertexVersion(executionVertexId, currentVersion); } diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java index 97b4b24f8ac..83fb50f1514 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java @@ -21,6 +21,7 @@ package org.apache.flink.runtime.scheduler.adaptivebatch; import org.apache.flink.annotation.VisibleForTesting; import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.JobStatus; import org.apache.flink.api.common.time.Time; import org.apache.flink.configuration.Configuration; import org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint; @@ -62,6 +63,7 @@ import org.apache.flink.runtime.scheduler.DefaultScheduler; import org.apache.flink.runtime.scheduler.ExecutionGraphFactory; import org.apache.flink.runtime.scheduler.ExecutionOperations; import org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorFactory; +import org.apache.flink.runtime.scheduler.ExecutionVertexVersion; import org.apache.flink.runtime.scheduler.ExecutionVertexVersioner; import org.apache.flink.runtime.scheduler.VertexParallelismStore; import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID; @@ -187,8 +189,10 @@ public class AdaptiveBatchScheduler extends DefaultScheduler { protected void startSchedulingInternal() { tryComputeSourceParallelismThenRunAsync( (Void value, Throwable throwable) -> { -initializeVerticesIfPossible(); -super.startSchedulingInternal(); +if (getExecutionGraph().getState() == JobStatus.CREATED) { +initializeVerticesIfPossible(); +super.startSchedulingInternal(); +} }); } @@ -196,8 +200,16 @@ public class AdaptiveBatchScheduler extends DefaultScheduler { protected void onTaskFinished(final Execution execution, final IOMetrics ioMetrics) { checkNotNull(ioMetrics); updateResultPartitionBytesMetrics(ioMetrics.getResultPartitionBytes()); +ExecutionVertexVersion currentVersion = + executionVertexVersioner.getExecutionVertexVersion(execution.getVertex().getID()); tryComputeSourceParallelismThenRunAsync( (Void value, Throwable throwable) -> { +if (executionVertexVersioner.isModified(currentVersion)) { +log.debug( +"Initialization of vertices will be skipped, because the execution" ++ " vertex version has been modified."); +return; +} initializeVerticesIfPossible(); super.onTaskFinished(execution, ioMetrics);
(flink) branch master updated: [FLINK-34126][configuration] Correct the description of config option jobmanager.scheduler
This is an automated email from the ASF dual-hosted git repository. zhuzh pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 2be1ea801cf [FLINK-34126][configuration] Correct the description of config option jobmanager.scheduler 2be1ea801cf is described below commit 2be1ea801cf616d0d0a82729829245c205caaad8 Author: JunRuiLee AuthorDate: Wed Jan 17 13:54:55 2024 +0800 [FLINK-34126][configuration] Correct the description of config option jobmanager.scheduler This closes #24112. --- .../generated/all_jobmanager_section.html | 2 +- .../generated/expert_scheduling_section.html | 2 +- .../generated/job_manager_configuration.html | 2 +- .../flink/configuration/JobManagerOptions.java | 51 ++ 4 files changed, 36 insertions(+), 21 deletions(-) diff --git a/docs/layouts/shortcodes/generated/all_jobmanager_section.html b/docs/layouts/shortcodes/generated/all_jobmanager_section.html index 79b0e7f804b..626aa50f386 100644 --- a/docs/layouts/shortcodes/generated/all_jobmanager_section.html +++ b/docs/layouts/shortcodes/generated/all_jobmanager_section.html @@ -120,7 +120,7 @@ jobmanager.scheduler Default Enum -Determines which scheduler implementation is used to schedule tasks. Accepted values are:'Default': Default scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch': Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here jobstore.cache-size diff --git a/docs/layouts/shortcodes/generated/expert_scheduling_section.html b/docs/layouts/shortcodes/generated/expert_scheduling_section.html index 56837bda5e3..0c9c877c844 100644 --- a/docs/layouts/shortcodes/generated/expert_scheduling_section.html +++ b/docs/layouts/shortcodes/generated/expert_scheduling_section.html @@ -102,7 +102,7 @@ jobmanager.scheduler Default Enum -Determines which scheduler implementation is used to schedule tasks. Accepted values are:'Default': Default scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch': Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here scheduler-mode diff --git a/docs/layouts/shortcodes/generated/job_manager_configuration.html b/docs/layouts/shortcodes/generated/job_manager_configuration.html index d07705d7018..b82ae03c3d7 100644 --- a/docs/layouts/shortcodes/generated/job_manager_configuration.html +++ b/docs/layouts/shortcodes/generated/job_manager_configuration.html @@ -174,7 +174,7 @@ jobmanager.scheduler Default Enum -Determines which scheduler implementation is used to schedule tasks. Accepted values are:'Default': Default scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch': Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here jobstore.cache-size diff --git a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java index 0378e380923..1e6c68c3506 100644 --- a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java +++ b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java @@ -18,9 +18,11 @@ package org.apache.flink.configuration; +import org.apache.flink.annotation.Internal; import org.apache.flink.annotation.PublicEvolving; import org.apache.flink.annotation.docs.Documentation; import org.apache.flink.configuration.description.Description; +import org.apache.flink.configuration.description.InlineElement; import java.time.Duration; @@ -458,29 +460,42 @@ public class JobManagerOptions { .withDescription( Description.builder() .text( -"Determines which scheduler implementation is used to schedule tasks. Accepted values are:") -.list( -text("'Default': Default scheduler"), -text( -"'Adaptive': Adaptive scheduler. More details can be found %s.", -link( - "{{.Site.BaseURL}}{{.Site.LanguagePrefix}}/docs/deployment/elastic_scaling#adaptive-scheduler", -"here")), -