(flink-connector-pulsar) branch main updated: [hotfix] Update copyright year to 2024
This is an automated email from the ASF dual-hosted git repository. leonard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-pulsar.git The following commit(s) were added to refs/heads/main by this push: new 5889eb2 [hotfix] Update copyright year to 2024 5889eb2 is described below commit 5889eb21d13126f0b4e616cabb6f07365188215e Author: Hang Ruan AuthorDate: Wed Jan 31 09:51:57 2024 +0800 [hotfix] Update copyright year to 2024 This closes #82. --- flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE b/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE index 4730a99..de3d655 100644 --- a/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE +++ b/flink-sql-connector-pulsar/src/main/resources/META-INF/NOTICE @@ -1,5 +1,5 @@ flink-sql-connector-pulsar -Copyright 2014-2023 The Apache Software Foundation +Copyright 2014-2024 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
(flink-connector-hbase) branch main updated (42870d5 -> 91d166d)
This is an automated email from the ASF dual-hosted git repository. leonard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-hbase.git from 42870d5 [hotfix] Remove 1.19-SNAPSHOT since HBase currently can't compile for that version add 91d166d [hotfix] Update copyright year to 2024 No new revisions were added by this update. Summary of changes: flink-sql-connector-hbase-1.4/src/main/resources/META-INF/NOTICE | 2 +- flink-sql-connector-hbase-2.2/src/main/resources/META-INF/NOTICE | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(flink-connector-gcp-pubsub) branch main updated: [hotfix][ci] Make CI against multiple Flink versions
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
The following commit(s) were added to refs/heads/main by this push:
new 4c155b4 [hotfix][ci] Make CI against multiple Flink versions
4c155b4 is described below
commit 4c155b49e41d157d780a2d1e80c388017b0a2941
Author: Hang Ruan
AuthorDate: Wed Jan 31 11:49:47 2024 +0800
[hotfix][ci] Make CI against multiple Flink versions
This closes #21.
---
.github/workflows/push_pr.yml | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml
index 335f108..99cf8d7 100644
--- a/.github/workflows/push_pr.yml
+++ b/.github/workflows/push_pr.yml
@@ -23,6 +23,9 @@ concurrency:
cancel-in-progress: true
jobs:
compile_and_test:
+strategy:
+ matrix:
+flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT, 1.18-SNAPSHOT]
uses: apache/flink-connector-shared-utils/.github/workflows/ci.yml@ci_utils
with:
- flink_version: 1.17.0
+ flink_version: ${{ matrix.flink }}
(flink-connector-elasticsearch) branch main updated (153b8fc -> 4914929)
This is an automated email from the ASF dual-hosted git repository. leonard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-elasticsearch.git from 153b8fc [hotfix] Set maximum supported Flink version to 1.17 for `v3.0` add 7bda67e [FLINK-34113][elasticsearch] Make flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces This closes #88. add 4914929 [hotfix] Add dependency convergence for org.yaml:snakeyaml No new revisions were added by this update. Summary of changes: flink-connector-elasticsearch-base/pom.xml | 8 ++ .../sink/ElasticsearchSinkBuilderBaseTest.java | 100 + flink-connector-elasticsearch6/pom.xml | 8 ++ flink-connector-elasticsearch7/pom.xml | 8 ++ .../src/main/resources/META-INF/NOTICE | 2 + pom.xml| 7 ++ 6 files changed, 36 insertions(+), 97 deletions(-)
(flink-connector-elasticsearch) branch dependabot/maven/org.yaml-snakeyaml-2.0 created (now 60c726d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.yaml-snakeyaml-2.0 in repository https://gitbox.apache.org/repos/asf/flink-connector-elasticsearch.git at 60c726d Bump org.yaml:snakeyaml from 1.31 to 2.0 No new revisions were added by this update.
(flink-connector-jdbc) branch main updated (5a90eb0a -> 611b45cf)
This is an automated email from the ASF dual-hosted git repository. guoweijie pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-jdbc.git from 5a90eb0a [FLINK-33365] include filters with Lookup joins add 611b45cf [FLINK-34259] Creating ExecutionConfig instead of mocking it in tests No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/jdbc/JdbcTestBase.java | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
(flink) branch master updated (0ba3e76b147 -> b5a2ee4e988)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 0ba3e76b147 [docs] Fix typo in ide_setup.md (#20877) add b5a2ee4e988 [FLINK-34286][k8s] Attach cluster config map labels at creation time No new revisions were added by this update. Summary of changes: .../KubernetesLeaderElectionDriver.java| 10 -- .../resources/KubernetesLeaderElector.java | 16 +++-- .../KubernetesLeaderElectionDriverTest.java| 23 .../resources/KubernetesLeaderElectorITCase.java | 42 ++ 4 files changed, 56 insertions(+), 35 deletions(-)
(flink) branch master updated (b5a2ee4e988 -> 85edd784fc7)
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from b5a2ee4e988 [FLINK-34286][k8s] Attach cluster config map labels at creation time new 1f23e2ce6f1 [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found new f10702b8c23 [hotfix][ci] Fixes condition for filtering NOTICE files new adde027cae6 [hotfix][build] Adds Maven and Java version to Maven command new edfdc89a029 [hotfix][ci] Replaces spaces with tabs new e4b4718009a [hotfix][ci] Adds proper log message at the end of a successful CI run new f4c0b716b01 [FLINK-34062][build] Adds JDK 21 surefire configuration. new a810088688c [FLINK-27082][ci] Updates github-actions profile. new 85edd784fc7 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow The 8 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/actions/job_init/action.yml| 72 .github/actions/run_mvn/action.yml | 42 +++ .../workflows/ci.yml | 56 ++- .github/workflows/template.flink-ci.yml| 386 + .github/workflows/template.pre-compile-checks.yml | 77 pom.xml| 51 ++- tools/azure-pipelines/create_build_artifact.sh | 12 +- tools/azure-pipelines/debug_files_utils.sh | 62 ++-- tools/azure-pipelines/unpack_build_artifact.sh | 20 +- tools/azure-pipelines/uploading_watchdog.sh| 4 +- .../tools/ci/licensecheck/NoticeFileChecker.java | 3 +- tools/ci/maven-utils.sh| 1 + 12 files changed, 716 insertions(+), 70 deletions(-) create mode 100644 .github/actions/job_init/action.yml create mode 100644 .github/actions/run_mvn/action.yml copy tools/releasing/deploy_staging_jars.sh => .github/workflows/ci.yml (52%) mode change 100755 => 100644 create mode 100644 .github/workflows/template.flink-ci.yml create mode 100644 .github/workflows/template.pre-compile-checks.yml
(flink) 03/08: [hotfix][build] Adds Maven and Java version to Maven command
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit adde027cae657909d4e4cc6411f6b65e67970c1b
Author: Matthias Pohl
AuthorDate: Tue Jan 16 16:20:45 2024 +0100
[hotfix][build] Adds Maven and Java version to Maven command
---
tools/ci/maven-utils.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/ci/maven-utils.sh b/tools/ci/maven-utils.sh
index 2ac16179b23..ab535af4320 100755
--- a/tools/ci/maven-utils.sh
+++ b/tools/ci/maven-utils.sh
@@ -17,6 +17,7 @@
# Utility for invoking Maven in CI
function run_mvn {
if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then
+ $MAVEN_WRAPPER --version
echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'"
fi
$MAVEN_WRAPPER $MVN_GLOBAL_OPTIONS "${@}"
(flink) 01/08: [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 1f23e2ce6f1ef9295d6899034f005467d8c94607
Author: Matthias Pohl
AuthorDate: Thu Dec 21 18:27:46 2023 +0100
[hotfix][ci] Fixes find commands where touch might cause a failure if no
files were found
---
tools/azure-pipelines/create_build_artifact.sh | 2 +-
tools/azure-pipelines/unpack_build_artifact.sh | 11 +--
2 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/tools/azure-pipelines/create_build_artifact.sh
b/tools/azure-pipelines/create_build_artifact.sh
index 809d14fe651..0beeca3b075 100755
--- a/tools/azure-pipelines/create_build_artifact.sh
+++ b/tools/azure-pipelines/create_build_artifact.sh
@@ -28,7 +28,7 @@ echo "Minimizing artifact files"
# by removing files not required for subsequent stages
# jars are re-built in subsequent stages, so no need to cache them (cannot be
avoided)
-find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' | xargs rm -rf
+find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' -exec rm -rf {} \;
# .git directory
# not deleting this can cause build stability issues
diff --git a/tools/azure-pipelines/unpack_build_artifact.sh
b/tools/azure-pipelines/unpack_build_artifact.sh
index 15d5c7c1057..68f8d221aae 100755
--- a/tools/azure-pipelines/unpack_build_artifact.sh
+++ b/tools/azure-pipelines/unpack_build_artifact.sh
@@ -28,15 +28,14 @@ cp -RT "$FLINK_ARTIFACT_DIR" "."
echo "Adjusting timestamps"
# adjust timestamps of proto file to avoid re-generation
-find . -type f -name '*.proto' | xargs touch
+find . -type f -name '*.proto' -exec touch {} \;
# wait a bit for better odds of different timestamps
sleep 5
# adjust timestamps to prevent recompilation
-find . -type f -name '*.java' | xargs touch
-find . -type f -name '*.scala' | xargs touch
+find . -type f -name '*.java' -exec touch {} \;
+find . -type f -name '*.scala' -exec touch {} \;
# wait a bit for better odds of different timestamps
sleep 5
-find . -type f -name '*.class' | xargs touch
-find . -type f -name '*.timestamp' | xargs touch
-
+find . -type f -name '*.class' -exec touch {} \;
+find . -type f -name '*.timestamp' -exec touch {} \;
(flink) 08/08: [FLINK-33914][ci] Introduces a basic GitHub Actions workflow
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 85edd784fc72c1784849e2b122cbf3215f89817c
Author: Matthias Pohl
AuthorDate: Sun Dec 10 13:36:57 2023 +0100
[FLINK-33914][ci] Introduces a basic GitHub Actions workflow
- Adds basic CI workflow
- Adds composite workflow for starting Flink's CI stages
- Updates CI helper scripts to support GitHub Actions
- Adds composite workflow for doing pre-compile checks
- Adds custom action to enable maven-utils.sh in workflows
- Adds custom action for initializing a job
---
.github/actions/job_init/action.yml | 72
.github/actions/run_mvn/action.yml| 42 +++
.github/workflows/ci.yml | 40 +++
.github/workflows/template.flink-ci.yml | 386 ++
.github/workflows/template.pre-compile-checks.yml | 77 +
tools/azure-pipelines/create_build_artifact.sh| 10 +-
tools/azure-pipelines/debug_files_utils.sh| 18 +
tools/azure-pipelines/unpack_build_artifact.sh| 9 +-
tools/azure-pipelines/uploading_watchdog.sh | 2 +
9 files changed, 653 insertions(+), 3 deletions(-)
diff --git a/.github/actions/job_init/action.yml
b/.github/actions/job_init/action.yml
new file mode 100644
index 000..5158d855857
--- /dev/null
+++ b/.github/actions/job_init/action.yml
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
+echo "The job's start time is set to ${job_start_time}."
+
+- name: "Set JDK version to ${{ inputs.jdk_version }}"
+ shell: bash
+ run: |
+echo "JAVA_HOME=$JAVA_HOME_${{ inputs.jdk_version }}_X64" >>
"${GITHUB_ENV}"
+echo "PATH=$JAVA_HOME_${{ inputs.jdk_version }}_X64/bin:$PATH" >>
"${GITHUB_ENV}"
+
+- name: "Setup Maven package cache"
+ if: ${{ inputs.maven_repo_folder != '' }}
+ uses: actions/cache@v4
+ with:
+path: ${{ inputs.maven_repo_folder }}
+key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+restore-keys: ${{ runner.os }}-maven
+
+- name: "Moves checkout content from ${{ inputs.source_directory }} to ${{
inputs.target_directory }}."
+ if: ${{ inputs.source_directory != '' && inputs.target_directory != '' }}
+ shell: bash
+ run: |
+mkdir -p ${{ inputs.target_directory }}
+
+# .scalafmt.conf is needed for Scala format checks
+# .mvn is needed to make the Maven wrapper accessible in test runs
+mv ${{ inputs.source_directory }}/* \
+ ${{ inputs.source_directory }}/.scalafmt.conf \
+ ${{ inputs.source_directory }}/.mvn \
+ ${{ inputs.target_directory }}
diff --git a/.github/actions/run_mvn/action.yml
b/.github/actions/run_mvn/action.yml
new file mode 100644
index 000..03595a25cdb
--- /dev/null
+++ b/.github/actions/run_mvn/action.yml
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
(flink) 02/08: [hotfix][ci] Fixes condition for filtering NOTICE files
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit f10702b8c235a3e4fb25f7efc89da5d775e74b2d
Author: Matthias Pohl
AuthorDate: Fri Dec 8 17:17:05 2023 +0100
[hotfix][ci] Fixes condition for filtering NOTICE files
---
.../java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
index 9a8b10db810..e39bcda1c50 100644
---
a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
+++
b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
@@ -359,7 +359,8 @@ public class NoticeFileChecker {
.filter(
file -> {
int nameCount = file.getNameCount();
-return file.getName(nameCount -
3).toString().equals("resources")
+return nameCount >= 3
+&& file.getName(nameCount -
3).toString().equals("resources")
&& file.getName(nameCount -
2).toString().equals("META-INF")
&& file.getName(nameCount -
1).toString().equals("NOTICE");
})
(flink) 05/08: [hotfix][ci] Adds proper log message at the end of a successful CI run
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit e4b4718009a2237fcb4b77df3573c9c96d747ce7
Author: Matthias Pohl
AuthorDate: Wed Jan 24 10:31:30 2024 +0100
[hotfix][ci] Adds proper log message at the end of a successful CI run
---
tools/azure-pipelines/uploading_watchdog.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/azure-pipelines/uploading_watchdog.sh
b/tools/azure-pipelines/uploading_watchdog.sh
index 2eafc1dfcbe..dc3d227ecbe 100755
--- a/tools/azure-pipelines/uploading_watchdog.sh
+++ b/tools/azure-pipelines/uploading_watchdog.sh
@@ -105,8 +105,8 @@ WATCHDOG_PID=$!
( $COMMAND & PID=$! ; echo $PID >$MAIN_PID_FILE ; wait $PID ) | ts | tee
$DEBUG_FILES_OUTPUT_DIR/watchdog
TEST_EXIT_CODE=${PIPESTATUS[0]}
-# successful execution, cleanup watchdog related things
if [[ "$TEST_EXIT_CODE" == 0 ]]; then
+ echo "[INFO] Test execution finished with exit code 0. Artifacts related to
the watchdog process will be cleaned up."
kill $WATCHDOG_PID
rm $DEBUG_FILES_OUTPUT_DIR/watchdog
rm -f $DEBUG_FILES_OUTPUT_DIR/jps-traces.*
(flink) 07/08: [FLINK-27082][ci] Updates github-actions profile.
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a810088688c7fe1bc969ed99aed3aaaf73742449
Author: Matthias Pohl
AuthorDate: Fri Jan 5 15:48:03 2024 +0100
[FLINK-27082][ci] Updates github-actions profile.
A dedicated github-actions profile doesn't work if other profiles which
configure excludeGroups for the surefire plugin are enabled.
---
pom.xml | 46 ++
1 file changed, 38 insertions(+), 8 deletions(-)
diff --git a/pom.xml b/pom.xml
index f7de4a97b5b..9c20d806d77 100644
--- a/pom.xml
+++ b/pom.xml
@@ -204,6 +204,10 @@ under the License.
of entries in the form
'[-]{2}add-[opens|exports]=/=ALL-UNNAMED'.-->
+
+
+
+
**/*Test.*
@@ -1000,6 +1004,9 @@ under the License.
enable-adaptive-scheduler
+
+
org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler
+
@@ -1009,7 +1016,11 @@ under the License.
true
-
org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk,
+
@@ -1018,7 +1029,9 @@ under the License.
github-actions
-
+
+
org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser
+
@@ -1026,7 +1039,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk},
+
@@ -1040,6 +1057,10 @@ under the License.
[11,)
+
+
org.apache.flink.testutils.junit.FailsOnJava11
+
+
@@ -1052,7 +1073,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsOnJava11
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk},
+
@@ -1101,6 +1126,7 @@ under the License.
(flink) 06/08: [FLINK-34062][build] Adds JDK 21 surefire configuration.
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit f4c0b716b01200d5e02bb49d9d859934867080a4
Author: Matthias Pohl
AuthorDate: Thu Jan 11 12:50:38 2024 +0100
[FLINK-34062][build] Adds JDK 21 surefire configuration.
---
pom.xml | 11 +++
1 file changed, 11 insertions(+)
diff --git a/pom.xml b/pom.xml
index fbf34fcd46b..f7de4a97b5b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1169,6 +1169,17 @@ under the License.
true
+
+
org.apache.maven.plugins
+
maven-surefire-plugin
+
+
+
${github-actions.surefire.excludedGroups},
+
org.apache.flink.testutils.junit.FailsOnJava11,
+
org.apache.flink.testutils.junit.FailsOnJava17,
+
+
+
(flink) 04/08: [hotfix][ci] Replaces spaces with tabs
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit edfdc89a029fce0fa87fcec08eaefb0fc3745c9f
Author: Matthias Pohl
AuthorDate: Wed Jan 24 10:12:19 2024 +0100
[hotfix][ci] Replaces spaces with tabs
The space indentation was accidentally added in FLINK-33906
---
tools/azure-pipelines/debug_files_utils.sh | 44 +++---
1 file changed, 22 insertions(+), 22 deletions(-)
diff --git a/tools/azure-pipelines/debug_files_utils.sh
b/tools/azure-pipelines/debug_files_utils.sh
index 6b74912e9fe..c2b3c3d5e8e 100755
--- a/tools/azure-pipelines/debug_files_utils.sh
+++ b/tools/azure-pipelines/debug_files_utils.sh
@@ -30,28 +30,28 @@ function prepare_debug_files {
export DEBUG_FILES_OUTPUT_DIR="${parent_directory}/debug_files"
export DEBUG_FILES_NAME="$(echo "${module}" | tr -c '[:alnum:]\n\r'
'_')-$(date +%s)"
- if [ -n "${TF_BUILD+x}" ]; then
-echo "[INFO] Azure Pipelines environment detected: $0 will export the
variables in the Azure-specific way."
-
-echo "##vso[task.setvariable
variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR"
-echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME"
- elif [ -n "${GITHUB_ACTIONS+x}" ]; then
-echo "[INFO] GitHub Actions environment detected: $0 will export the
variables in the GHA-specific way."
-
-if [ -z "${GITHUB_OUTPUT+x}" ]; then
- echo "[ERROR] The GITHUB_OUTPUT variable is not set."
- exit 1
-elif [ ! -f "$GITHUB_OUTPUT" ]; then
- echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file:
$GITHUB_OUTPUT"
- exit 1
-fi
-
-echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT"
-echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT"
- else
-echo "[ERROR] No CI environment detected. Debug artifact-related variables
couldn't be exported."
-exit 1
- fi
+ if [ -n "${TF_BUILD+x}" ]; then
+ echo "[INFO] Azure Pipelines environment detected: $0 will
export the variables in the Azure-specific way."
+
+ echo "##vso[task.setvariable
variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR"
+ echo "##vso[task.setvariable
variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME"
+ elif [ -n "${GITHUB_ACTIONS+x}" ]; then
+ echo "[INFO] GitHub Actions environment detected: $0 will
export the variables in the GHA-specific way."
+
+ if [ -z "${GITHUB_OUTPUT+x}" ]; then
+ echo "[ERROR] The GITHUB_OUTPUT variable is not set."
+ exit 1
+ elif [ ! -f "$GITHUB_OUTPUT" ]; then
+ echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer
to a file: $GITHUB_OUTPUT"
+ exit 1
+ fi
+
+ echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >>
"$GITHUB_OUTPUT"
+ echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT"
+ else
+ echo "[ERROR] No CI environment detected. Debug
artifact-related variables couldn't be exported."
+ exit 1
+ fi
mkdir -p $DEBUG_FILES_OUTPUT_DIR || { echo "FAILURE: cannot create
debug files directory '${DEBUG_FILES_OUTPUT_DIR}'." ; exit 1; }
}
(flink) 06/07: [FLINK-27082][ci] Updates github-actions profile.
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit fff63f4443e2f98ccd49f6cdbb46a28e52371d38
Author: Matthias Pohl
AuthorDate: Fri Jan 5 15:48:03 2024 +0100
[FLINK-27082][ci] Updates github-actions profile.
A dedicated github-actions profile doesn't work if other profiles which
configure excludeGroups for the surefire plugin are enabled.
---
pom.xml | 40 +++-
1 file changed, 35 insertions(+), 5 deletions(-)
diff --git a/pom.xml b/pom.xml
index 8547171216a..3246cf9a81a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -202,6 +202,10 @@ under the License.
of entries in the form
'[-]{2}add-[opens|exports]=/=ALL-UNNAMED'.-->
+
+
+
+
**/*Test.*
@@ -1000,6 +1004,9 @@ under the License.
enable-adaptive-scheduler
+
+
org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler
+
@@ -1009,7 +1016,11 @@ under the License.
true
-
org.apache.flink.testutils.junit.FailsWithAdaptiveScheduler
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk,
+
@@ -1018,7 +1029,9 @@ under the License.
github-actions
-
+
+
org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser
+
@@ -1026,7 +1039,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsInGHAContainerWithRootUser
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk},
+
@@ -1040,6 +1057,10 @@ under the License.
[11,)
+
+
org.apache.flink.testutils.junit.FailsOnJava11
+
+
@@ -1052,7 +1073,11 @@ under the License.
org.apache.maven.plugins
maven-surefire-plugin
-
org.apache.flink.testutils.junit.FailsOnJava11
+
+
${surefire.excludedGroups.github-actions},
+
${surefire.excludedGroups.adaptive-scheduler},
+
${surefire.excludedGroups.jdk},
+
@@ -1101,6 +1126,7 @@ under the License.
(flink) branch release-1.18 updated (d903990e1ed -> a64e1c98533)
This is an automated email from the ASF dual-hosted git repository. mapohl pushed a change to branch release-1.18 in repository https://gitbox.apache.org/repos/asf/flink.git from d903990e1ed [FLINK-34115][table-planner] Fix TableAggregateITCase unstable test new 2c6493bcc1e [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found new 227f50b2b66 [hotfix][ci] Fixes condition for filtering NOTICE files new ffdce2cef6a [hotfix][build] Adds Maven and Java version to Maven command new bfa590d1b52 [hotfix][ci] Replaces spaces with tabs new 4da0f1b05eb [hotfix][ci] Adds proper log message at the end of a successful CI run new fff63f4443e [FLINK-27082][ci] Updates github-actions profile. new a64e1c98533 [FLINK-33914][ci] Introduces a basic GitHub Actions workflow The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/actions/job_init/action.yml| 72 .github/actions/run_mvn/action.yml | 42 +++ .../workflows/ci.yml | 57 ++- .github/workflows/template.flink-ci.yml| 386 + .github/workflows/template.pre-compile-checks.yml | 77 pom.xml| 40 ++- tools/azure-pipelines/create_build_artifact.sh | 12 +- tools/azure-pipelines/debug_files_utils.sh | 62 ++-- tools/azure-pipelines/unpack_build_artifact.sh | 20 +- tools/azure-pipelines/uploading_watchdog.sh| 4 +- .../tools/ci/licensecheck/NoticeFileChecker.java | 3 +- tools/ci/maven-utils.sh| 1 + 12 files changed, 705 insertions(+), 71 deletions(-) create mode 100644 .github/actions/job_init/action.yml create mode 100644 .github/actions/run_mvn/action.yml copy tools/releasing/deploy_staging_jars.sh => .github/workflows/ci.yml (52%) mode change 100755 => 100644 create mode 100644 .github/workflows/template.flink-ci.yml create mode 100644 .github/workflows/template.pre-compile-checks.yml
(flink) 01/07: [hotfix][ci] Fixes find commands where touch might cause a failure if no files were found
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 2c6493bcc1ef643330b59bea77bc5f3d703f9d85
Author: Matthias Pohl
AuthorDate: Thu Dec 21 18:27:46 2023 +0100
[hotfix][ci] Fixes find commands where touch might cause a failure if no
files were found
---
tools/azure-pipelines/create_build_artifact.sh | 2 +-
tools/azure-pipelines/unpack_build_artifact.sh | 11 +--
2 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/tools/azure-pipelines/create_build_artifact.sh
b/tools/azure-pipelines/create_build_artifact.sh
index 809d14fe651..0beeca3b075 100755
--- a/tools/azure-pipelines/create_build_artifact.sh
+++ b/tools/azure-pipelines/create_build_artifact.sh
@@ -28,7 +28,7 @@ echo "Minimizing artifact files"
# by removing files not required for subsequent stages
# jars are re-built in subsequent stages, so no need to cache them (cannot be
avoided)
-find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' | xargs rm -rf
+find "$FLINK_ARTIFACT_DIR" -maxdepth 8 -type f -name '*.jar' -exec rm -rf {} \;
# .git directory
# not deleting this can cause build stability issues
diff --git a/tools/azure-pipelines/unpack_build_artifact.sh
b/tools/azure-pipelines/unpack_build_artifact.sh
index 15d5c7c1057..68f8d221aae 100755
--- a/tools/azure-pipelines/unpack_build_artifact.sh
+++ b/tools/azure-pipelines/unpack_build_artifact.sh
@@ -28,15 +28,14 @@ cp -RT "$FLINK_ARTIFACT_DIR" "."
echo "Adjusting timestamps"
# adjust timestamps of proto file to avoid re-generation
-find . -type f -name '*.proto' | xargs touch
+find . -type f -name '*.proto' -exec touch {} \;
# wait a bit for better odds of different timestamps
sleep 5
# adjust timestamps to prevent recompilation
-find . -type f -name '*.java' | xargs touch
-find . -type f -name '*.scala' | xargs touch
+find . -type f -name '*.java' -exec touch {} \;
+find . -type f -name '*.scala' -exec touch {} \;
# wait a bit for better odds of different timestamps
sleep 5
-find . -type f -name '*.class' | xargs touch
-find . -type f -name '*.timestamp' | xargs touch
-
+find . -type f -name '*.class' -exec touch {} \;
+find . -type f -name '*.timestamp' -exec touch {} \;
(flink) 04/07: [hotfix][ci] Replaces spaces with tabs
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit bfa590d1b52fd70baf1918821f5cd8bd219e8785
Author: Matthias Pohl
AuthorDate: Wed Jan 24 10:12:19 2024 +0100
[hotfix][ci] Replaces spaces with tabs
The space indentation was accidentally added in FLINK-33906
---
tools/azure-pipelines/debug_files_utils.sh | 44 +++---
1 file changed, 22 insertions(+), 22 deletions(-)
diff --git a/tools/azure-pipelines/debug_files_utils.sh
b/tools/azure-pipelines/debug_files_utils.sh
index 6b74912e9fe..c2b3c3d5e8e 100755
--- a/tools/azure-pipelines/debug_files_utils.sh
+++ b/tools/azure-pipelines/debug_files_utils.sh
@@ -30,28 +30,28 @@ function prepare_debug_files {
export DEBUG_FILES_OUTPUT_DIR="${parent_directory}/debug_files"
export DEBUG_FILES_NAME="$(echo "${module}" | tr -c '[:alnum:]\n\r'
'_')-$(date +%s)"
- if [ -n "${TF_BUILD+x}" ]; then
-echo "[INFO] Azure Pipelines environment detected: $0 will export the
variables in the Azure-specific way."
-
-echo "##vso[task.setvariable
variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR"
-echo "##vso[task.setvariable variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME"
- elif [ -n "${GITHUB_ACTIONS+x}" ]; then
-echo "[INFO] GitHub Actions environment detected: $0 will export the
variables in the GHA-specific way."
-
-if [ -z "${GITHUB_OUTPUT+x}" ]; then
- echo "[ERROR] The GITHUB_OUTPUT variable is not set."
- exit 1
-elif [ ! -f "$GITHUB_OUTPUT" ]; then
- echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer to a file:
$GITHUB_OUTPUT"
- exit 1
-fi
-
-echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >> "$GITHUB_OUTPUT"
-echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT"
- else
-echo "[ERROR] No CI environment detected. Debug artifact-related variables
couldn't be exported."
-exit 1
- fi
+ if [ -n "${TF_BUILD+x}" ]; then
+ echo "[INFO] Azure Pipelines environment detected: $0 will
export the variables in the Azure-specific way."
+
+ echo "##vso[task.setvariable
variable=DEBUG_FILES_OUTPUT_DIR]$DEBUG_FILES_OUTPUT_DIR"
+ echo "##vso[task.setvariable
variable=DEBUG_FILES_NAME]$DEBUG_FILES_NAME"
+ elif [ -n "${GITHUB_ACTIONS+x}" ]; then
+ echo "[INFO] GitHub Actions environment detected: $0 will
export the variables in the GHA-specific way."
+
+ if [ -z "${GITHUB_OUTPUT+x}" ]; then
+ echo "[ERROR] The GITHUB_OUTPUT variable is not set."
+ exit 1
+ elif [ ! -f "$GITHUB_OUTPUT" ]; then
+ echo "[ERROR] The GITHUB_OUTPUT variable doesn't refer
to a file: $GITHUB_OUTPUT"
+ exit 1
+ fi
+
+ echo "debug-files-output-dir=${DEBUG_FILES_OUTPUT_DIR}" >>
"$GITHUB_OUTPUT"
+ echo "debug-files-name=${DEBUG_FILES_NAME}" >> "$GITHUB_OUTPUT"
+ else
+ echo "[ERROR] No CI environment detected. Debug
artifact-related variables couldn't be exported."
+ exit 1
+ fi
mkdir -p $DEBUG_FILES_OUTPUT_DIR || { echo "FAILURE: cannot create
debug files directory '${DEBUG_FILES_OUTPUT_DIR}'." ; exit 1; }
}
(flink) 05/07: [hotfix][ci] Adds proper log message at the end of a successful CI run
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 4da0f1b05ebd192db58b41886d396fa68eca6e01
Author: Matthias Pohl
AuthorDate: Wed Jan 24 10:31:30 2024 +0100
[hotfix][ci] Adds proper log message at the end of a successful CI run
---
tools/azure-pipelines/uploading_watchdog.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/azure-pipelines/uploading_watchdog.sh
b/tools/azure-pipelines/uploading_watchdog.sh
index 2eafc1dfcbe..dc3d227ecbe 100755
--- a/tools/azure-pipelines/uploading_watchdog.sh
+++ b/tools/azure-pipelines/uploading_watchdog.sh
@@ -105,8 +105,8 @@ WATCHDOG_PID=$!
( $COMMAND & PID=$! ; echo $PID >$MAIN_PID_FILE ; wait $PID ) | ts | tee
$DEBUG_FILES_OUTPUT_DIR/watchdog
TEST_EXIT_CODE=${PIPESTATUS[0]}
-# successful execution, cleanup watchdog related things
if [[ "$TEST_EXIT_CODE" == 0 ]]; then
+ echo "[INFO] Test execution finished with exit code 0. Artifacts related to
the watchdog process will be cleaned up."
kill $WATCHDOG_PID
rm $DEBUG_FILES_OUTPUT_DIR/watchdog
rm -f $DEBUG_FILES_OUTPUT_DIR/jps-traces.*
(flink) 02/07: [hotfix][ci] Fixes condition for filtering NOTICE files
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 227f50b2b66ec8963d637d8929088aca91d88c04
Author: Matthias Pohl
AuthorDate: Fri Dec 8 17:17:05 2023 +0100
[hotfix][ci] Fixes condition for filtering NOTICE files
---
.../java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
index 9a8b10db810..e39bcda1c50 100644
---
a/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
+++
b/tools/ci/flink-ci-tools/src/main/java/org/apache/flink/tools/ci/licensecheck/NoticeFileChecker.java
@@ -359,7 +359,8 @@ public class NoticeFileChecker {
.filter(
file -> {
int nameCount = file.getNameCount();
-return file.getName(nameCount -
3).toString().equals("resources")
+return nameCount >= 3
+&& file.getName(nameCount -
3).toString().equals("resources")
&& file.getName(nameCount -
2).toString().equals("META-INF")
&& file.getName(nameCount -
1).toString().equals("NOTICE");
})
(flink) 07/07: [FLINK-33914][ci] Introduces a basic GitHub Actions workflow
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a64e1c98533649d8567f416fb94432d93ddf1d28
Author: Matthias Pohl
AuthorDate: Sun Dec 10 13:36:57 2023 +0100
[FLINK-33914][ci] Introduces a basic GitHub Actions workflow
- Adds basic CI workflow
- Adds composite workflow for starting Flink's CI stages
- Updates CI helper scripts to support GitHub Actions
- Adds composite workflow for doing pre-compile checks
- Adds custom action to enable maven-utils.sh in workflows
- Adds custom action for initializing a job
---
.github/actions/job_init/action.yml | 72
.github/actions/run_mvn/action.yml| 42 +++
.github/workflows/ci.yml | 40 +++
.github/workflows/template.flink-ci.yml | 386 ++
.github/workflows/template.pre-compile-checks.yml | 77 +
tools/azure-pipelines/create_build_artifact.sh| 10 +-
tools/azure-pipelines/debug_files_utils.sh| 18 +
tools/azure-pipelines/unpack_build_artifact.sh| 9 +-
tools/azure-pipelines/uploading_watchdog.sh | 2 +
9 files changed, 653 insertions(+), 3 deletions(-)
diff --git a/.github/actions/job_init/action.yml
b/.github/actions/job_init/action.yml
new file mode 100644
index 000..5158d855857
--- /dev/null
+++ b/.github/actions/job_init/action.yml
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Initializes the job"
+description: "Does all the necessary steps to set up the job"
+inputs:
+ jdk_version:
+description: "The JDK version that's supposed to be used."
+required: true
+ maven_repo_folder:
+description: "The location of the local Maven repository (not setting this
parameter will omit the caching of Maven artifacts)."
+required: false
+default: ""
+ source_directory:
+description: "Specifies the directory from which the code should be moved
from (needed for containerized runs; not setting this parameter will omit
moving the checkout)."
+required: false
+default: ""
+ target_directory:
+description: "Specifies the directory to which the code should be moved to
(needed for containerized runs; not setting this parameter will omit moving the
checkout)."
+required: false
+default: ""
+runs:
+ using: "composite"
+ steps:
+- name: "Initializes GHA_PIPELINE_START_TIME with the job's start time"
+ shell: bash
+ run: |
+job_start_time="$(date --rfc-3339=seconds)"
+echo "GHA_PIPELINE_START_TIME=${job_start_time}" >> "${GITHUB_ENV}"
+echo "The job's start time is set to ${job_start_time}."
+
+- name: "Set JDK version to ${{ inputs.jdk_version }}"
+ shell: bash
+ run: |
+echo "JAVA_HOME=$JAVA_HOME_${{ inputs.jdk_version }}_X64" >>
"${GITHUB_ENV}"
+echo "PATH=$JAVA_HOME_${{ inputs.jdk_version }}_X64/bin:$PATH" >>
"${GITHUB_ENV}"
+
+- name: "Setup Maven package cache"
+ if: ${{ inputs.maven_repo_folder != '' }}
+ uses: actions/cache@v4
+ with:
+path: ${{ inputs.maven_repo_folder }}
+key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+restore-keys: ${{ runner.os }}-maven
+
+- name: "Moves checkout content from ${{ inputs.source_directory }} to ${{
inputs.target_directory }}."
+ if: ${{ inputs.source_directory != '' && inputs.target_directory != '' }}
+ shell: bash
+ run: |
+mkdir -p ${{ inputs.target_directory }}
+
+# .scalafmt.conf is needed for Scala format checks
+# .mvn is needed to make the Maven wrapper accessible in test runs
+mv ${{ inputs.source_directory }}/* \
+ ${{ inputs.source_directory }}/.scalafmt.conf \
+ ${{ inputs.source_directory }}/.mvn \
+ ${{ inputs.target_directory }}
diff --git a/.github/actions/run_mvn/action.yml
b/.github/actions/run_mvn/action.yml
new file mode 100644
index 000..03595a25cdb
--- /dev/null
+++ b/.github/actions/run_mvn/action.yml
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more co
(flink) 03/07: [hotfix][build] Adds Maven and Java version to Maven command
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
commit ffdce2cef6a56db77be209d5721c2f6fd3208eea
Author: Matthias Pohl
AuthorDate: Tue Jan 16 16:20:45 2024 +0100
[hotfix][build] Adds Maven and Java version to Maven command
---
tools/ci/maven-utils.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/ci/maven-utils.sh b/tools/ci/maven-utils.sh
index 2ac16179b23..ab535af4320 100755
--- a/tools/ci/maven-utils.sh
+++ b/tools/ci/maven-utils.sh
@@ -17,6 +17,7 @@
# Utility for invoking Maven in CI
function run_mvn {
if [[ "$MVN_RUN_VERBOSE" != "false" ]]; then
+ $MAVEN_WRAPPER --version
echo "Invoking mvn with '$MVN_GLOBAL_OPTIONS ${@}'"
fi
$MAVEN_WRAPPER $MVN_GLOBAL_OPTIONS "${@}"
(flink) branch master updated: [FLINK-33915][ci] Adds workflow for extended master CI workflow
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 697d328f6bb [FLINK-33915][ci] Adds workflow for extended master CI
workflow
697d328f6bb is described below
commit 697d328f6bb090db63b80f063d3298446a33a5b9
Author: Matthias Pohl
AuthorDate: Fri Dec 15 10:46:13 2023 +0100
[FLINK-33915][ci] Adds workflow for extended master CI workflow
---
.github/workflows/nightly.yml | 98 +++
1 file changed, 98 insertions(+)
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
new file mode 100644
index 000..4f2603a6c27
--- /dev/null
+++ b/.github/workflows/nightly.yml
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow is meant as an extended CI run that includes certain features
that shall be tested
+# and JDK versions that are supported but not considered default.
+
+name: "Nightly (beta)"
+
+on:
+ schedule:
+- cron: '0 2 * * *'
+ workflow_dispatch:
+
+permissions: read-all
+
+jobs:
+ pre-compile-checks:
+name: "Pre-compile Checks"
+uses: ./.github/workflows/template.pre-compile-checks.yml
+
+ java8:
+name: "Java 8"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java8
+ environment: 'PROFILE="-Dinclude_hadoop_aws"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ java11:
+name: "Java 11"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java11
+ environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
+ jdk_version: 11
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ java17:
+name: "Java 17"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java17
+ environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
+ jdk_version: 17
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ java21:
+name: "Java 21"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java21
+ environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21
-Pjava21-target"'
+ jdk_version: 21
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ hadoop313:
+name: "Hadoop 3.1.3"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: hadoop313
+ environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ adaptive-scheduler:
+name: "AdaptiveScheduler"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: adaptive-scheduler
+ environment: 'PROFILE="-Penable-adaptive-scheduler"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
(flink) branch release-1.18 updated: [FLINK-33915][ci] Adds workflow release-1.18 nightly CI workflow
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch release-1.18
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.18 by this push:
new a27e03650d9 [FLINK-33915][ci] Adds workflow release-1.18 nightly CI
workflow
a27e03650d9 is described below
commit a27e03650d9e03c9317e19051f8aa4871f199f4a
Author: Matthias Pohl
AuthorDate: Fri Dec 15 10:46:13 2023 +0100
[FLINK-33915][ci] Adds workflow release-1.18 nightly CI workflow
---
.github/workflows/nightly.yml | 85 +++
1 file changed, 85 insertions(+)
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
new file mode 100644
index 000..c02da2e4954
--- /dev/null
+++ b/.github/workflows/nightly.yml
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This workflow is meant as an extended CI run that includes certain features
that shall be tested
+# and JDK versions that are supported but not considered default.
+
+name: "Nightly (beta)"
+
+on:
+ workflow_dispatch:
+
+permissions: read-all
+
+jobs:
+ pre-compile-checks:
+name: "Pre-compile Checks"
+uses: ./.github/workflows/template.pre-compile-checks.yml
+
+ java8:
+name: "Java 8"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java8
+ environment: 'PROFILE="-Dinclude_hadoop_aws"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ java11:
+name: "Java 11"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java11
+ environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
+ jdk_version: 11
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ java17:
+name: "Java 17"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: java17
+ environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
+ jdk_version: 17
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ hadoop313:
+name: "Hadoop 3.1.3"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: hadoop313
+ environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ adaptive-scheduler:
+name: "AdaptiveScheduler"
+uses: ./.github/workflows/template.flink-ci.yml
+with:
+ workflow-caller-id: adaptive-scheduler
+ environment: 'PROFILE="-Penable-adaptive-scheduler"'
+ jdk_version: 8
+secrets:
+ s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
+ s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
+ s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
(flink) branch master updated (697d328f6bb -> ff9a3c85f69)
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 697d328f6bb [FLINK-33915][ci] Adds workflow for extended master CI
workflow
add ff9a3c85f69 [FLINK-33916][ci] Adds workflow trigger
No new revisions were added by this update.
Summary of changes:
.github/workflows/{ci.yml => nightly-trigger.yml} | 45 +--
.github/workflows/nightly.yml | 2 -
2 files changed, 25 insertions(+), 22 deletions(-)
copy .github/workflows/{ci.yml => nightly-trigger.yml} (51%)
(flink) branch master updated: [FLINK-34316] Reduce instantiation of ScanRuntimeProvider in streaming mode
This is an automated email from the ASF dual-hosted git repository.
twalthr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 503580834e0 [FLINK-34316] Reduce instantiation of ScanRuntimeProvider
in streaming mode
503580834e0 is described below
commit 503580834e06455bbcaf619a51f8d18812bbe83f
Author: Timo Walther
AuthorDate: Wed Jan 31 09:34:48 2024 +0100
[FLINK-34316] Reduce instantiation of ScanRuntimeProvider in streaming mode
---
.../apache/flink/table/planner/connectors/DynamicSourceUtils.java | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java
index 925d1a00b11..2ac8aeafe75 100644
---
a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java
+++
b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/connectors/DynamicSourceUtils.java
@@ -498,14 +498,12 @@ public final class DynamicSourceUtils {
ScanTableSource scanSource,
boolean isBatchMode,
ReadableConfig config) {
-final ScanRuntimeProvider provider =
-
scanSource.getScanRuntimeProvider(ScanRuntimeProviderContext.INSTANCE);
final ChangelogMode changelogMode = scanSource.getChangelogMode();
validateWatermarks(tableDebugName, schema);
if (isBatchMode) {
-validateScanSourceForBatch(tableDebugName, changelogMode,
provider);
+validateScanSourceForBatch(tableDebugName, scanSource,
changelogMode);
} else {
validateScanSourceForStreaming(
tableDebugName, schema, scanSource, changelogMode, config);
@@ -558,7 +556,9 @@ public final class DynamicSourceUtils {
}
private static void validateScanSourceForBatch(
-String tableDebugName, ChangelogMode changelogMode,
ScanRuntimeProvider provider) {
+String tableDebugName, ScanTableSource scanSource, ChangelogMode
changelogMode) {
+final ScanRuntimeProvider provider =
+
scanSource.getScanRuntimeProvider(ScanRuntimeProviderContext.INSTANCE);
// batch only supports bounded source
if (!provider.isBounded()) {
throw new ValidationException(
(flink) branch master updated: [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT
This is an automated email from the ASF dual-hosted git repository.
jchan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new cb9e220c229 [FLINK-34271][table-planner] Fix unstable
GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT
cb9e220c229 is described below
commit cb9e220c2291088459f0281aa8e8e8584436a9b2
Author: Xuyang
AuthorDate: Wed Jan 31 21:32:43 2024 +0800
[FLINK-34271][table-planner] Fix unstable
GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT
This closes #24226
---
.../exec/stream/GroupAggregateTestPrograms.java | 10 ++
.../plan/agg-with-state-ttl-hint.json | 2 +-
.../agg-with-state-ttl-hint/savepoint/_metadata | Bin 11151 -> 11151 bytes
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java
index 6c43cf90959..755eb781a35 100644
---
a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java
+++
b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/GroupAggregateTestPrograms.java
@@ -387,13 +387,15 @@ public class GroupAggregateTestPrograms {
"+I[1, 1, null, Hi]",
"+I[2, 1, 2.0, Hello]",
"+U[2, 2, 2.0, Hello]")
+// Due to state TTL in hint, the state in
the metadata
+// savepoint has expired.
.consumedAfterRestore(
-"+U[1, 2, null, Hi]",
-"+U[2, 3, 2.0, Hello]",
-"+U[2, 4, 2.0, Hello]")
+"+I[1, 1, null, Hi Again!]",
+"+I[2, 1, 2.0, Hello Again!]",
+"+U[2, 2, 2.0, Hello Again!]")
.build())
.runSql(
-"INSERT INTO sink_t SELECT /*+
STATE_TTL('source_t' = '4d') */"
+"INSERT INTO sink_t SELECT /*+
STATE_TTL('source_t' = '1s') */"
+ "b, "
+ "COUNT(*) AS cnt, "
+ "AVG(a) FILTER (WHERE a > 1) AS avg_a, "
diff --git
a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json
b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json
index cebe1e5898b..cfdcfed459b 100644
---
a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json
+++
b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/plan/agg-with-state-ttl-hint.json
@@ -128,7 +128,7 @@
"needRetraction" : false,
"state" : [ {
"index" : 0,
- "ttl" : "4 d",
+ "ttl" : "1 s",
"name" : "groupAggregateState"
} ],
"inputProperties" : [ {
diff --git
a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata
b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata
index 50e4bee3a94..e956dafb981 100644
Binary files
a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata
and
b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-group-aggregate_1/agg-with-state-ttl-hint/savepoint/_metadata
differ
(flink-kubernetes-operator) branch main updated: [FLINK-34311] Do not change min resource requirements when rescaling for adaptive scheduler
This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 4342636c [FLINK-34311] Do not change min resource requirements when
rescaling for adaptive scheduler
4342636c is described below
commit 4342636cdb2c3439389e83cb4fe4366156edfbd7
Author: Gyula Fora
AuthorDate: Tue Jan 30 15:43:46 2024 +0100
[FLINK-34311] Do not change min resource requirements when rescaling for
adaptive scheduler
---
.../kubernetes/operator/service/NativeFlinkService.java | 12 +---
.../kubernetes/operator/service/NativeFlinkServiceTest.java | 10 +++---
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java
index 9781a5b0..41083d91 100644
---
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java
+++
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/NativeFlinkService.java
@@ -200,9 +200,15 @@ public class NativeFlinkService extends
AbstractFlinkService {
var overrideStr = newOverrides.get(jobId);
if (overrideStr != null) {
-// We have an override for the vertex
-int p = Integer.parseInt(overrideStr);
-var newParallelism = new
JobVertexResourceRequirements.Parallelism(p, p);
+// We set the parallelism upper bound to the target
parallelism, anything higher
+// would defeat the purpose of scaling down
+int upperBound = Integer.parseInt(overrideStr);
+// We only change the lower bound if the new parallelism
went below it. As we
+// cannot guarantee that new resources can be acquired,
increasing the lower
+// bound to the target could potentially cause job failure.
+int lowerBound = Math.min(upperBound,
parallelism.getLowerBound());
+var newParallelism =
+new
JobVertexResourceRequirements.Parallelism(lowerBound, upperBound);
// If the requirements changed we mark this as scaling
triggered
if (!parallelism.equals(newParallelism)) {
entry.setValue(new
JobVertexResourceRequirements(newParallelism));
diff --git
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java
index c8c0afb3..693a9019 100644
---
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java
+++
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/service/NativeFlinkServiceTest.java
@@ -229,7 +229,9 @@ public class NativeFlinkServiceTest {
var reconStatus = flinkDep.getStatus().getReconciliationStatus();
reconStatus.serializeAndSetLastReconciledSpec(spec, flinkDep);
-appConfig.set(PipelineOptions.PARALLELISM_OVERRIDES,
Map.of(v1.toHexString(), "4"));
+appConfig.set(
+PipelineOptions.PARALLELISM_OVERRIDES,
+Map.of(v1.toHexString(), "4", v2.toHexString(), "1"));
spec.setFlinkConfiguration(appConfig.toMap());
flinkDep.getStatus().getJobStatus().setState("RUNNING");
@@ -256,13 +258,15 @@ public class NativeFlinkServiceTest {
Map.of(
v1,
new JobVertexResourceRequirements(
-new
JobVertexResourceRequirements.Parallelism(4, 4)),
+new
JobVertexResourceRequirements.Parallelism(1, 4)),
v2,
new JobVertexResourceRequirements(
-new
JobVertexResourceRequirements.Parallelism(2, 2))),
+new
JobVertexResourceRequirements.Parallelism(1, 1))),
updated.get());
// Baseline
+appConfig.set(PipelineOptions.PARALLELISM_OVERRIDES,
Map.of(v1.toHexString(), "4"));
+spec.setFlinkConfiguration(appConfig.toMap());
testScaleConditionDep(
flinkDep, service, d -> {},
FlinkService.ScalingResult.SCALING_TRIGGERED);
testScaleConditionLastSpec(
(flink-connector-jdbc) branch v3.1 updated: [hotfix] Use 1.16.x as minimum supported version in pom
This is an automated email from the ASF dual-hosted git repository. snuyanzin pushed a commit to branch v3.1 in repository https://gitbox.apache.org/repos/asf/flink-connector-jdbc.git The following commit(s) were added to refs/heads/v3.1 by this push: new 390d7bc9 [hotfix] Use 1.16.x as minimum supported version in pom 390d7bc9 is described below commit 390d7bc9139204fbfc48fe275a69eb60c4807fb5 Author: Sergey Nuyanzin AuthorDate: Wed Jan 31 16:44:11 2024 +0100 [hotfix] Use 1.16.x as minimum supported version in pom --- .github/workflows/push_pr.yml | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml index a54f4a0d..38724aef 100644 --- a/.github/workflows/push_pr.yml +++ b/.github/workflows/push_pr.yml @@ -25,7 +25,7 @@ jobs: compile_and_test: strategy: matrix: -flink: [1.17-SNAPSHOT] +flink: [ 1.16-SNAPSHOT, 1.17-SNAPSHOT ] jdk: [ '8, 11' ] include: - flink: 1.18-SNAPSHOT diff --git a/pom.xml b/pom.xml index dd645500..9d44d460 100644 --- a/pom.xml +++ b/pom.xml @@ -47,7 +47,7 @@ under the License. -1.17.0 +1.16.0 2.13.4.20221013 4.13.2
(flink) branch master updated (cb9e220c229 -> 8e11b6005c9)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from cb9e220c229 [FLINK-34271][table-planner] Fix unstable GroupAggregateRestoreTest#AGG_WITH_STATE_TTL_HINT add 8e11b6005c9 [FLINK-34272][runtime] ExecutingState should never transition during its construction. No new revisions were added by this update. Summary of changes: .../runtime/scheduler/adaptive/Executing.java | 2 +- .../runtime/scheduler/adaptive/ExecutingTest.java | 109 + .../MockStateWithExecutionGraphContext.java| 5 +- 3 files changed, 50 insertions(+), 66 deletions(-)
(flink) branch master updated (8e11b6005c9 -> 98e1d079f6e)
This is an automated email from the ASF dual-hosted git repository. libenchao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 8e11b6005c9 [FLINK-34272][runtime] ExecutingState should never transition during its construction. add 3c8088b716a [hotfix][table-common] Mark SinkV2Provider as the recommended core interface for SinkRuntimeProvider add 98e1d079f6e [FLINK-34270][docs] Update doc for the new Table source & sink interfaces introduced in FLIP-146/367 No new revisions were added by this update. Summary of changes: docs/content.zh/docs/dev/table/sourcesSinks.md | 12 ++-- docs/content/docs/dev/table/sourcesSinks.md| 22 ++ .../table/connector/sink/DynamicTableSink.java | 16 3 files changed, 36 insertions(+), 14 deletions(-)
(flink) branch master updated: [FLINK-33358][sql-client] Fix Flink SQL Client fail to start in Flink on YARN (#23629)
This is an automated email from the ASF dual-hosted git repository.
shengkai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new fa364c7c668 [FLINK-33358][sql-client] Fix Flink SQL Client fail to
start in Flink on YARN (#23629)
fa364c7c668 is described below
commit fa364c7c668ec2a87bcbf18ce2b80f749cc16b2b
Author: Prabhu Joseph
AuthorDate: Thu Feb 1 09:21:40 2024 +0530
[FLINK-33358][sql-client] Fix Flink SQL Client fail to start in Flink on
YARN (#23629)
Co-authored-by: Prabhu Joseph
---
flink-table/flink-sql-client/bin/sql-client.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-table/flink-sql-client/bin/sql-client.sh
b/flink-table/flink-sql-client/bin/sql-client.sh
index 3091bdf1b36..797dcc4ac05 100755
--- a/flink-table/flink-sql-client/bin/sql-client.sh
+++ b/flink-table/flink-sql-client/bin/sql-client.sh
@@ -87,7 +87,7 @@ if [[ "$CC_CLASSPATH" =~ .*flink-sql-client.*.jar ]]; then
elif [ -n "$FLINK_SQL_CLIENT_JAR" ]; then
# start client with jar
-exec "$JAVA_RUN" $FLINK_ENV_JAVA_OPTS $JVM_ARGS "${log_setting[@]}"
-classpath "`manglePathList
"$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS:$FLINK_SQL_CLIENT_JAR"`"
org.apache.flink.table.client.SqlClient "$@" --jar "`manglePath
$FLINK_SQL_CLIENT_JAR`"
+exec "$JAVA_RUN" $FLINK_ENV_JAVA_OPTS $JVM_ARGS "${log_setting[@]}"
-classpath "`manglePathList
"$CC_CLASSPATH:$FLINK_SQL_CLIENT_JAR:$INTERNAL_HADOOP_CLASSPATHS"`"
org.apache.flink.table.client.SqlClient "$@" --jar "`manglePath
$FLINK_SQL_CLIENT_JAR`"
# write error message to stderr
else
(flink) branch master updated: [FLINK-34206][runtime] Fix potential job failure due to concurrent global failure and source paralelism inference
This is an automated email from the ASF dual-hosted git repository.
zhuzh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new b737b718596 [FLINK-34206][runtime] Fix potential job failure due to
concurrent global failure and source paralelism inference
b737b718596 is described below
commit b737b71859672e8020881ce2abf998735ee98abb
Author: sunxia
AuthorDate: Tue Jan 30 14:26:26 2024 +0800
[FLINK-34206][runtime] Fix potential job failure due to concurrent global
failure and source paralelism inference
This closes #24223.
---
.../runtime/scheduler/ExecutionVertexVersioner.java | 2 +-
.../scheduler/adaptivebatch/AdaptiveBatchScheduler.java | 16 ++--
.../apache/flink/test/streaming/runtime/CacheITCase.java | 2 --
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java
index b0a0b17db0d..a86d22fc40c 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionVertexVersioner.java
@@ -92,7 +92,7 @@ public class ExecutionVertexVersioner {
ExecutionVertexVersion::getExecutionVertexId,
Function.identity()));
}
-ExecutionVertexVersion getExecutionVertexVersion(ExecutionVertexID
executionVertexId) {
+public ExecutionVertexVersion getExecutionVertexVersion(ExecutionVertexID
executionVertexId) {
final long currentVersion = getCurrentVersion(executionVertexId);
return new ExecutionVertexVersion(executionVertexId, currentVersion);
}
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java
index 97b4b24f8ac..83fb50f1514 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java
@@ -21,6 +21,7 @@ package org.apache.flink.runtime.scheduler.adaptivebatch;
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.configuration.Configuration;
import
org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint;
@@ -62,6 +63,7 @@ import org.apache.flink.runtime.scheduler.DefaultScheduler;
import org.apache.flink.runtime.scheduler.ExecutionGraphFactory;
import org.apache.flink.runtime.scheduler.ExecutionOperations;
import org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorFactory;
+import org.apache.flink.runtime.scheduler.ExecutionVertexVersion;
import org.apache.flink.runtime.scheduler.ExecutionVertexVersioner;
import org.apache.flink.runtime.scheduler.VertexParallelismStore;
import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
@@ -187,8 +189,10 @@ public class AdaptiveBatchScheduler extends
DefaultScheduler {
protected void startSchedulingInternal() {
tryComputeSourceParallelismThenRunAsync(
(Void value, Throwable throwable) -> {
-initializeVerticesIfPossible();
-super.startSchedulingInternal();
+if (getExecutionGraph().getState() == JobStatus.CREATED) {
+initializeVerticesIfPossible();
+super.startSchedulingInternal();
+}
});
}
@@ -196,8 +200,16 @@ public class AdaptiveBatchScheduler extends
DefaultScheduler {
protected void onTaskFinished(final Execution execution, final IOMetrics
ioMetrics) {
checkNotNull(ioMetrics);
updateResultPartitionBytesMetrics(ioMetrics.getResultPartitionBytes());
+ExecutionVertexVersion currentVersion =
+
executionVertexVersioner.getExecutionVertexVersion(execution.getVertex().getID());
tryComputeSourceParallelismThenRunAsync(
(Void value, Throwable throwable) -> {
+if (executionVertexVersioner.isModified(currentVersion)) {
+log.debug(
+"Initialization of vertices will be skipped,
because the execution"
++ " vertex version has been
modified.");
+return;
+}
initializeVerticesIfPossible();
super.onTaskFinished(execution, ioMetrics);
(flink) branch master updated: [FLINK-34126][configuration] Correct the description of config option jobmanager.scheduler
This is an automated email from the ASF dual-hosted git repository.
zhuzh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 2be1ea801cf [FLINK-34126][configuration] Correct the description of
config option jobmanager.scheduler
2be1ea801cf is described below
commit 2be1ea801cf616d0d0a82729829245c205caaad8
Author: JunRuiLee
AuthorDate: Wed Jan 17 13:54:55 2024 +0800
[FLINK-34126][configuration] Correct the description of config option
jobmanager.scheduler
This closes #24112.
---
.../generated/all_jobmanager_section.html | 2 +-
.../generated/expert_scheduling_section.html | 2 +-
.../generated/job_manager_configuration.html | 2 +-
.../flink/configuration/JobManagerOptions.java | 51 ++
4 files changed, 36 insertions(+), 21 deletions(-)
diff --git a/docs/layouts/shortcodes/generated/all_jobmanager_section.html
b/docs/layouts/shortcodes/generated/all_jobmanager_section.html
index 79b0e7f804b..626aa50f386 100644
--- a/docs/layouts/shortcodes/generated/all_jobmanager_section.html
+++ b/docs/layouts/shortcodes/generated/all_jobmanager_section.html
@@ -120,7 +120,7 @@
jobmanager.scheduler
Default
Enum
-Determines which scheduler implementation is used to schedule
tasks. Accepted values are:'Default': Default
scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch':
Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here
jobstore.cache-size
diff --git a/docs/layouts/shortcodes/generated/expert_scheduling_section.html
b/docs/layouts/shortcodes/generated/expert_scheduling_section.html
index 56837bda5e3..0c9c877c844 100644
--- a/docs/layouts/shortcodes/generated/expert_scheduling_section.html
+++ b/docs/layouts/shortcodes/generated/expert_scheduling_section.html
@@ -102,7 +102,7 @@
jobmanager.scheduler
Default
Enum
-Determines which scheduler implementation is used to schedule
tasks. Accepted values are:'Default': Default
scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch':
Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here
scheduler-mode
diff --git a/docs/layouts/shortcodes/generated/job_manager_configuration.html
b/docs/layouts/shortcodes/generated/job_manager_configuration.html
index d07705d7018..b82ae03c3d7 100644
--- a/docs/layouts/shortcodes/generated/job_manager_configuration.html
+++ b/docs/layouts/shortcodes/generated/job_manager_configuration.html
@@ -174,7 +174,7 @@
jobmanager.scheduler
Default
Enum
-Determines which scheduler implementation is used to schedule
tasks. Accepted values are:'Default': Default
scheduler'Adaptive': Adaptive scheduler. More details can be found here.'AdaptiveBatch':
Adaptive batch scheduler. More details can be found "Adaptive": Adaptive scheduler. More details can be found here
jobstore.cache-size
diff --git
a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
index 0378e380923..1e6c68c3506 100644
---
a/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
+++
b/flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java
@@ -18,9 +18,11 @@
package org.apache.flink.configuration;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.annotation.docs.Documentation;
import org.apache.flink.configuration.description.Description;
+import org.apache.flink.configuration.description.InlineElement;
import java.time.Duration;
@@ -458,29 +460,42 @@ public class JobManagerOptions {
.withDescription(
Description.builder()
.text(
-"Determines which scheduler
implementation is used to schedule tasks. Accepted values are:")
-.list(
-text("'Default': Default
scheduler"),
-text(
-"'Adaptive': Adaptive
scheduler. More details can be found %s.",
-link(
-
"{{.Site.BaseURL}}{{.Site.LanguagePrefix}}/docs/deployment/elastic_scaling#adaptive-scheduler",
-"here")),
-
