svn commit: r61953 - in /dev/flink/flink-1.16.2-rc1: ./ python/
Author: guoweijie Date: Thu May 18 06:37:46 2023 New Revision: 61953 Log: Add flink-1.16.2-rc1 Added: dev/flink/flink-1.16.2-rc1/ dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz (with props) dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz.asc dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz.sha512 dev/flink/flink-1.16.2-rc1/flink-1.16.2-src.tgz (with props) dev/flink/flink-1.16.2-rc1/flink-1.16.2-src.tgz.asc dev/flink/flink-1.16.2-rc1/flink-1.16.2-src.tgz.sha512 dev/flink/flink-1.16.2-rc1/python/ dev/flink/flink-1.16.2-rc1/python/apache-flink-1.16.2.tar.gz (with props) dev/flink/flink-1.16.2-rc1/python/apache-flink-1.16.2.tar.gz.asc dev/flink/flink-1.16.2-rc1/python/apache-flink-1.16.2.tar.gz.sha512 dev/flink/flink-1.16.2-rc1/python/apache-flink-libraries-1.16.2.tar.gz (with props) dev/flink/flink-1.16.2-rc1/python/apache-flink-libraries-1.16.2.tar.gz.asc dev/flink/flink-1.16.2-rc1/python/apache-flink-libraries-1.16.2.tar.gz.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-manylinux1_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-manylinux1_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp36-cp36m-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-manylinux1_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp37-cp37m-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_11_0_arm64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_11_0_arm64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-macosx_11_0_arm64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-manylinux1_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-manylinux1_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp38-cp38-manylinux1_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_10_9_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_10_9_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_10_9_x86_64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_11_0_arm64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_11_0_arm64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-macosx_11_0_arm64.whl.sha512 dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-manylinux1_x86_64.whl (with props) dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-manylinux1_x86_64.whl.asc dev/flink/flink-1.16.2-rc1/python/apache_flink-1.16.2-cp39-cp39-manylinux1_x86_64.whl.sha512 Added: dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz == Binary file - no diff available. Propchange: dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz -- svn:mime-type = application/octet-stream Added: dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz.asc == --- dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz.asc (added) +++ dev/flink/flink-1.16.2-rc1/flink-1.16.2-bin-scala_2.12.tgz.asc Thu May 18 06:37:46 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEjVaubnCCaZpIcHUOpOjEwF7mhh8FAmRlwO4ACgkQpOjEwF7m +hh/kPg//Y8uWH0eLglPRWuD37zPgCYRXVjOhsRmBSd8cwNsNyhJLbezAayrZ0iMy +4ic2HM/ekIYU3Ra/b8sHhJJj9BrscKSIrfOeNJqNGs8e8YTW2my/a
[flink-web] 01/02: Kubernetes Operator 1.5.0
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit db61e8dbbcc5a6fddcbe84e9a6f5218ee26817d9 Author: Gyula Fora AuthorDate: Thu May 11 14:45:31 2023 +0200 Kubernetes Operator 1.5.0 --- docs/config.toml | 4 +- ...2023-05-17-release-kubernetes-operator-1.5.0.md | 104 + docs/data/flink_kubernetes_operator.yml| 14 +-- docs/data/release_archive.yml | 3 + 4 files changed, 116 insertions(+), 9 deletions(-) diff --git a/docs/config.toml b/docs/config.toml index fb003fda8..26da58017 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -39,8 +39,8 @@ posts = "/:year/:month/:day/:title/" StateFunStableShortVersion = "3.2" FlinkMLStableVersion = "2.2.0" FlinkMLStableShortVersion = "2.2" - FlinkKubernetesOperatorStableVersion = "1.4.0" - FlinkKubernetesOperatorStableShortVersion = "1.4" + FlinkKubernetesOperatorStableVersion = "1.5.0" + FlinkKubernetesOperatorStableShortVersion = "1.5" FlinkTableStoreStableVersion = "0.3.0" FlinkTableStoreStableShortVersion = "0.3" diff --git a/docs/content/posts/2023-05-17-release-kubernetes-operator-1.5.0.md b/docs/content/posts/2023-05-17-release-kubernetes-operator-1.5.0.md new file mode 100644 index 0..98c511a74 --- /dev/null +++ b/docs/content/posts/2023-05-17-release-kubernetes-operator-1.5.0.md @@ -0,0 +1,104 @@ +--- +title: "Apache Flink Kubernetes Operator 1.5.0 Release Announcement" +date: "2023-05-17T08:00:00.000Z" +authors: +- gyfora: + name: "Gyula Fora" + twitter: "GyulaFora" +aliases: +- /news/2023/05/17/release-kubernetes-operator-1.5.0.html +--- + +The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.5.0! The release focuses on improvements to the job autoscaler that was introduced in the previous release and general operational hardening of the operator. + +We encourage you to [download the release](https://flink.apache.org/downloads.html) and share your feedback with the community through the Flink [mailing lists](https://flink.apache.org/community.html#mailing-lists) or [JIRA](https://issues.apache.org/jira/browse/flink)! We hope you like the new release and we’d be eager to learn about your experience with it. + +## Autoscaler improvements + +### Algorithm improvements and better scale down behaviour + +The release contains important improvements to the core autoscaling logic. This includes improved stability of scaling decisions (leading to less parallelism oscillations) and better handling of slow or idle streams. + +There are also some fixes related to output ratio computation and propagation that greatly improves the autoscaler on more complex streaming pipelines. + +This version also introduces new metrics for tracking the number of scaling decisions and scaling errors together with some more Kubernetes events to improve the observability of the system. + +### Improved default configuration + +We have simplified and improved some default autoscaler configs for a better out-of-the-box user experience. + +Some notable changes: + +``` +kubernetes.operator.job.autoscaler.metrics.window: 5m -> 10m +kubernetes.operator.job.autoscaler.target.utilization.boundary: 0.1 -> 0.4 +kubernetes.operator.job.autoscaler.scale-up.grace-period: 10m -> 1h + +kubernetes.operator.job.autoscaler.history.max.count: 1 -> 3 +kubernetes.operator.job.autoscaler.scaling.effectiveness.detection.enabled: true -> false + +kubernetes.operator.job.autoscaler.catch-up.duration: 10m -> 5m +kubernetes.operator.job.autoscaler.restart.time: 5m -> 3m +``` + +## CRD Changes + +### Ephemeral storage support + +Stateful streaming jobs often rely on ephemeral storage to store the working state of the pipeline. Previously it was only possible to change the ephemeral storage size through the pod template mechanism. The 1.5.0 release adds a new field to the task and jobmanager specification that allows configuring this similarly to other resources: + +``` +spec: + ... + taskManager: +resource: + memory: "2048m" + cpu: 8 + ephemeralStorage: "10G" +``` + +Make sure you upgrade the CRD together with the operator deployment to be able to access this feature. For more details check the [docs](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/upgrade/) + +## General operations + +### Fabric8 and JOSDK version bump + +The operator have been updated to use the latest Java Operator SDK and Fabric8 versions that contain important fixes for production environments. + +### Health probe and canary resources + +Previous operator versions already contained a rudimentary health probe to catch simple startup errors but did not have a good mechanism to catch errors that developed during the lifetime of the running operator. + +The 1.5.0
[flink-web] branch asf-site updated (702efc322 -> 1492c35cd)
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git from 702efc322 Rebuild website new db61e8dbb Kubernetes Operator 1.5.0 new 1492c35cd Rebuild website The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../08/26/apache-flink-0.6-available/index.html| 4 +- .../09/26/apache-flink-0.6.1-available/index.html | 4 +- content/2014/10/03/upcoming-events/index.html | 4 +- .../11/04/apache-flink-0.7.0-available/index.html | 4 +- .../11/18/hadoop-compatibility-in-flink/index.html | 4 +- .../index.html | 4 +- .../01/21/apache-flink-0.8.0-available/index.html | 4 +- .../january-2015-in-the-flink-community/index.html | 4 +- .../02/09/introducing-flink-streaming/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../march-2015-in-the-flink-community/index.html | 4 +- .../index.html | 4 +- .../05/11/juggling-with-bits-and-bytes/index.html | 4 +- .../april-2015-in-the-flink-community/index.html | 4 +- .../06/24/announcing-apache-flink-0.9.0/index.html | 4 +- .../index.html | 4 +- .../09/01/apache-flink-0.9.1-available/index.html | 4 +- .../09/03/announcing-flink-forward-2015/index.html | 4 +- .../index.html | 4 +- .../16/announcing-apache-flink-0.10.0/index.html | 4 +- .../2015/11/27/flink-0.10.1-released/index.html| 4 +- .../index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../2016/02/11/flink-0.10.2-released/index.html| 4 +- .../03/08/announcing-apache-flink-1.0.0/index.html | 4 +- content/2016/04/06/flink-1.0.1-released/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- content/2016/04/22/flink-1.0.2-released/index.html | 4 +- content/2016/05/11/flink-1.0.3-released/index.html | 4 +- .../index.html | 4 +- .../08/04/announcing-apache-flink-1.1.0/index.html | 4 +- content/2016/08/04/flink-1.1.1-released/index.html | 4 +- .../index.html | 4 +- .../09/05/apache-flink-1.1.2-released/index.html | 4 +- .../10/12/apache-flink-1.1.3-released/index.html | 4 +- .../apache-flink-in-2016-year-in-review/index.html | 4 +- .../12/21/apache-flink-1.1.4-released/index.html | 4 +- .../02/06/announcing-apache-flink-1.2.0/index.html | 4 +- .../03/23/apache-flink-1.1.5-released/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../04/26/apache-flink-1.2.1-released/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../06/23/apache-flink-1.3.1-released/index.html | 4 +- .../index.html | 4 +- .../08/05/apache-flink-1.3.2-released/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../apache-flink-in-2017-year-in-review/index.html | 4 +- .../index.html | 4 +- .../02/15/apache-flink-1.4.1-released/index.html | 4 +- .../index.html | 4 +- .../03/08/apache-flink-1.4.2-released/index.html | 4 +- .../03/15/apache-flink-1.3.3-released/index.html | 4 +- .../index.html | 4 +- .../07/12/apache-flink-1.5.1-released/index.html | 4 +- .../07/31/apache-flink-1.5.2-released/index.html | 4 +- .../index.html | 4 +- .../08/21/apache-flink-1.5.3-released/index.html | 4 +- .../09/20/apache-flink-1.5.4-released/index.html | 4 +- .../09/20/apache-flink-1.6.1-released/index.html | 4 +- .../10/29/apache-flink-1.5.5-released/index.html | 4 +- .../10/29/apache-flink-1.6.2-released/index.html | 4 +- .../index.html | 4 +- .../12/21/apache-flink-1.7.1-released/index.html | 4 +- .../12/22/apache-flink-1.6.3-released/index.html | 4 +- .../12/26/apache-flink-1.5.6-released/index.html | 4 +- .../index.html | 4 +- .../02/15/apache-flink-1.7.2-released/index.html | 4 +-
[flink] branch master updated: [hotfix] Fix typos of comment
This is an automated email from the ASF dual-hosted git repository. guoweijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 2c252922723 [hotfix] Fix typos of comment 2c252922723 is described below commit 2c25292272306a6eb094be40be0a0fa5fc3cf644 Author: Huston AuthorDate: Fri Apr 14 01:29:34 2023 +0800 [hotfix] Fix typos of comment --- .../generated/yarn_config_configuration.html | 2 +- .../flink/api/common/state/OperatorStateStore.java | 4 +-- .../apache/flink/optimizer/dag/OptimizerNode.java | 2 +- .../slotmanager/ResourceTracker.java | 2 +- .../checkpoint/CheckpointStatsCountsTest.java | 2 +- .../api/datastream/AllWindowedStream.java | 24 ++--- .../streaming/api/datastream/WindowedStream.java | 24 ++--- .../streaming/api/scala/AllWindowedStream.scala| 26 +++--- .../flink/streaming/api/scala/KeyedStream.scala| 40 +++--- .../flink/streaming/api/scala/WindowedStream.scala | 26 +++--- .../yarn/configuration/YarnConfigOptions.java | 2 +- 11 files changed, 75 insertions(+), 79 deletions(-) diff --git a/docs/layouts/shortcodes/generated/yarn_config_configuration.html b/docs/layouts/shortcodes/generated/yarn_config_configuration.html index 13037ab8575..77fe8dd978b 100644 --- a/docs/layouts/shortcodes/generated/yarn_config_configuration.html +++ b/docs/layouts/shortcodes/generated/yarn_config_configuration.html @@ -36,7 +36,7 @@ yarn.application-attempts (none) String -Number of ApplicationMaster restarts. By default, the value will be set to 1. If high availability is enabled, then the default value will be 2. The restart number is also limited by YARN (configured via https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml";>yarn.resourcemanager.am.max-attempts). Note that that the entire Flink cluster will restart and the YARN Client will lose the connection. +Number of ApplicationMaster restarts. By default, the value will be set to 1. If high availability is enabled, then the default value will be 2. The restart number is also limited by YARN (configured via https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml";>yarn.resourcemanager.am.max-attempts). Note that the entire Flink cluster will restart and the YARN Client will lose the connection. yarn.application-master.port diff --git a/flink-core/src/main/java/org/apache/flink/api/common/state/OperatorStateStore.java b/flink-core/src/main/java/org/apache/flink/api/common/state/OperatorStateStore.java index 4c24f09b834..cb005c64319 100644 --- a/flink-core/src/main/java/org/apache/flink/api/common/state/OperatorStateStore.java +++ b/flink-core/src/main/java/org/apache/flink/api/common/state/OperatorStateStore.java @@ -60,7 +60,7 @@ public interface OperatorStateStore { * * Note the semantic differences between an operator list state and a keyed list state (see * {@link KeyedStateStore#getListState(ListStateDescriptor)}). Under the context of operator - * state, the list is a collection of state items that are independent from each other and + * state, the list is a collection of state items that are independent of each other and * eligible for redistribution across operator instances in case of changed operator * parallelism. In other words, these state items are the finest granularity at which non-keyed * state can be redistributed, and should not be correlated with each other. @@ -83,7 +83,7 @@ public interface OperatorStateStore { * * Note the semantic differences between an operator list state and a keyed list state (see * {@link KeyedStateStore#getListState(ListStateDescriptor)}). Under the context of operator - * state, the list is a collection of state items that are independent from each other and + * state, the list is a collection of state items that are independent of each other and * eligible for redistribution across operator instances in case of changed operator * parallelism. In other words, these state items are the finest granularity at which non-keyed * state can be redistributed, and should not be correlated with each other. diff --git a/flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/OptimizerNode.java b/flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/OptimizerNode.java index 12b3955f9f1..a8f38c9f18b 100644 --- a/flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/OptimizerNode.java +++ b/flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/OptimizerNode.java @@ -878,7 +878,7 @@ public abstract class OptimizerNode // add the cheapest plan if (cheapest != null)
[flink] branch release-1.17 updated: [FLINK-31963][state] Fix rescaling bug in recovery from unaligned checkpoints. (#22584) (#22594)
This is an automated email from the ASF dual-hosted git repository. srichter pushed a commit to branch release-1.17 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.17 by this push: new 8d8a486aaa8 [FLINK-31963][state] Fix rescaling bug in recovery from unaligned checkpoints. (#22584) (#22594) 8d8a486aaa8 is described below commit 8d8a486aaa8360d6beabacc6980280c96bf900ea Author: Stefan Richter AuthorDate: Wed May 17 12:05:04 2023 +0200 [FLINK-31963][state] Fix rescaling bug in recovery from unaligned checkpoints. (#22584) (#22594) This commit fixes problems in StateAssignmentOperation for unaligned checkpoints with stateless operators that have upstream operators with output partition state or downstream operators with input channel state. (cherry picked from commit 354c0f455b92c083299d8028f161f0dd113ab614) --- .../checkpoint/StateAssignmentOperation.java | 28 ++-- .../runtime/checkpoint/TaskStateAssignment.java| 19 ++- .../checkpoint/StateAssignmentOperationTest.java | 178 - .../checkpointing/UnalignedCheckpointITCase.java | 18 ++- .../UnalignedCheckpointRescaleITCase.java | 137 ++-- .../checkpointing/UnalignedCheckpointTestBase.java | 32 +++- 6 files changed, 335 insertions(+), 77 deletions(-) diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java index 681e0b18df1..e476c6b65ec 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java @@ -136,19 +136,24 @@ public class StateAssignmentOperation { // repartition state for (TaskStateAssignment stateAssignment : vertexAssignments.values()) { -if (stateAssignment.hasNonFinishedState) { +if (stateAssignment.hasNonFinishedState +// FLINK-31963: We need to run repartitioning for stateless operators that have +// upstream output or downstream input states. +|| stateAssignment.hasUpstreamOutputStates() +|| stateAssignment.hasDownstreamInputStates()) { assignAttemptState(stateAssignment); } } // actually assign the state for (TaskStateAssignment stateAssignment : vertexAssignments.values()) { -// If upstream has output states, even the empty task state should be assigned for the -// current task in order to notify this task that the old states will send to it which -// likely should be filtered. +// If upstream has output states or downstream has input states, even the empty task +// state should be assigned for the current task in order to notify this task that the +// old states will send to it which likely should be filtered. if (stateAssignment.hasNonFinishedState || stateAssignment.isFullyFinished -|| stateAssignment.hasUpstreamOutputStates()) { +|| stateAssignment.hasUpstreamOutputStates() +|| stateAssignment.hasDownstreamInputStates()) { assignTaskStateToExecutionJobVertices(stateAssignment); } } @@ -345,9 +350,10 @@ public class StateAssignmentOperation { newParallelism))); } -public > void reDistributeResultSubpartitionStates( -TaskStateAssignment assignment) { -if (!assignment.hasOutputState) { +public void reDistributeResultSubpartitionStates(TaskStateAssignment assignment) { +// FLINK-31963: We can skip this phase if there is no output state AND downstream has no +// input states +if (!assignment.hasOutputState && !assignment.hasDownstreamInputStates()) { return; } @@ -394,7 +400,9 @@ public class StateAssignmentOperation { } public void reDistributeInputChannelStates(TaskStateAssignment stateAssignment) { -if (!stateAssignment.hasInputState) { +// FLINK-31963: We can skip this phase only if there is no input state AND upstream has no +// output states +if (!stateAssignment.hasInputState && !stateAssignment.hasUpstreamOutputStates()) { return; } @@ -435,7 +443,7 @@ public class StateAssignmentOperation { : getPartitionState( inputOperatorState, InputChannelInfo::getGateIdx, gateIndex); final MappingBasedRepartitioner repartitioner = -new MappingBasedRepartitioner(mapping); +new Mapping
[flink-kubernetes-operator] branch main updated: [hotfix][docs] fix some typos (#601)
This is an automated email from the ASF dual-hosted git repository. mxm pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git The following commit(s) were added to refs/heads/main by this push: new 47326397 [hotfix][docs] fix some typos (#601) 47326397 is described below commit 473263978e8be1f614f4d5951c5dd76299ffa698 Author: yangjf2019 <54518670+yangjf2...@users.noreply.github.com> AuthorDate: Wed May 17 18:04:59 2023 +0800 [hotfix][docs] fix some typos (#601) --- docs/content/docs/custom-resource/autoscaler.md | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/content/docs/custom-resource/autoscaler.md b/docs/content/docs/custom-resource/autoscaler.md index b1bdb797..748f9dc2 100644 --- a/docs/content/docs/custom-resource/autoscaler.md +++ b/docs/content/docs/custom-resource/autoscaler.md @@ -56,7 +56,7 @@ To disable scaling actions, set: `kubernetes.operator.job.autoscaler.scaling.ena Depending on your environment and job characteristics there are a few very important configurations that will affect how well the autoscaler works. -Key configuration areas +Key configuration areas: - Job and per operator max parallelism - Stabilization and metrics collection intervals - Target utilization and flexible boundaries @@ -67,9 +67,9 @@ The defaults might work reasonably well for many applications, but some tuning m ### Job and per operator max parallelism When computing the scaled parallelism, the autoscaler always considers the max parallelism settings for each job vertex to ensure that it doesn't introduce unnecessary data skew. -The computed parallelism will always be a divisor of the max_parallelism number. +The computed parallelism will always be a divisor of the max parallelism number. -To ensure flexible scaling it is therefore recommended to chose max parallelism settings that have a [lot of divisors](https://en.wikipedia.org/wiki/Highly_composite_number) instead of relying on the Flink provided defaults. +To ensure flexible scaling it is therefore recommended to choose max parallelism settings that have a [lot of divisors](https://en.wikipedia.org/wiki/Highly_composite_number) instead of relying on the Flink provided defaults. You can then use the `pipeline.max-parallelism` to configure this for your pipeline. Some good numbers for max-parallelism are: 120, 180, 240, 360, 720 etc. @@ -120,7 +120,6 @@ flinkConfiguration: kubernetes.operator.job.autoscaler.target.utilization.boundary: "0.2" kubernetes.operator.job.autoscaler.restart.time: 2m kubernetes.operator.job.autoscaler.catch-up.duration: 5m - pipeline.max-parallelism: "720" ```
[flink] branch master updated: [FLINK-31548][jdbc-driver] Introduce FlinkDataSource for flink jdbc driver
This is an automated email from the ASF dual-hosted git repository. libenchao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 420b5e319e1 [FLINK-31548][jdbc-driver] Introduce FlinkDataSource for flink jdbc driver 420b5e319e1 is described below commit 420b5e319e1f8e917c099530360f51f73a070575 Author: Shammon FY AuthorDate: Sat May 6 09:00:38 2023 +0800 [FLINK-31548][jdbc-driver] Introduce FlinkDataSource for flink jdbc driver Close apache/flink#22532 --- .../apache/flink/table/jdbc/FlinkDataSource.java | 88 ++ .../flink/table/jdbc/FlinkDataSourceTest.java | 39 ++ 2 files changed, 127 insertions(+) diff --git a/flink-table/flink-sql-jdbc-driver/src/main/java/org/apache/flink/table/jdbc/FlinkDataSource.java b/flink-table/flink-sql-jdbc-driver/src/main/java/org/apache/flink/table/jdbc/FlinkDataSource.java new file mode 100644 index 000..d8a84e19742 --- /dev/null +++ b/flink-table/flink-sql-jdbc-driver/src/main/java/org/apache/flink/table/jdbc/FlinkDataSource.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.jdbc; + +import javax.sql.DataSource; + +import java.io.PrintWriter; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.SQLFeatureNotSupportedException; +import java.util.Properties; +import java.util.logging.Logger; + +/** Basic flink data source which create {@link FlinkConnection}. */ +public class FlinkDataSource implements DataSource { +private final String url; +private final Properties properties; + +public FlinkDataSource(String url, Properties properties) { +this.url = url; +this.properties = properties; +} + +@Override +public Connection getConnection() throws SQLException { +return new FlinkConnection(DriverUri.create(url, properties)); +} + +@Override +public Connection getConnection(String username, String password) throws SQLException { +throw new SQLFeatureNotSupportedException( +"FlinkDataSource#getConnection with username and password is not supported"); +} + +@Override +public T unwrap(Class iface) throws SQLException { +throw new SQLFeatureNotSupportedException("FlinkDataSource#unwrap is not supported"); +} + +@Override +public boolean isWrapperFor(Class iface) throws SQLException { +throw new SQLFeatureNotSupportedException("FlinkDataSource#isWrapperFor is not supported"); +} + +@Override +public PrintWriter getLogWriter() throws SQLException { +throw new SQLFeatureNotSupportedException("FlinkDataSource#getLogWriter is not supported"); +} + +@Override +public void setLogWriter(PrintWriter out) throws SQLException { +throw new SQLFeatureNotSupportedException("FlinkDataSource#setLogWriter is not supported"); +} + +@Override +public void setLoginTimeout(int seconds) throws SQLException { +throw new SQLFeatureNotSupportedException( +"FlinkDataSource#setLoginTimeout is not supported"); +} + +@Override +public int getLoginTimeout() throws SQLException { +throw new SQLFeatureNotSupportedException( +"FlinkDataSource#getLoginTimeout is not supported"); +} + +@Override +public Logger getParentLogger() throws SQLFeatureNotSupportedException { +throw new SQLFeatureNotSupportedException( +"FlinkDataSource#getParentLogger is not supported"); +} +} diff --git a/flink-table/flink-sql-jdbc-driver/src/test/java/org/apache/flink/table/jdbc/FlinkDataSourceTest.java b/flink-table/flink-sql-jdbc-driver/src/test/java/org/apache/flink/table/jdbc/FlinkDataSourceTest.java new file mode 100644 index 000..9d00c37d24d --- /dev/null +++ b/flink-table/flink-sql-jdbc-driver/src/test/java/org/apache/flink/table/jdbc/FlinkDataSourceTest.java @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distribu
[flink-kubernetes-operator] annotated tag release-1.5.0-rc2 deleted (was 994792a5)
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a change to annotated tag release-1.5.0-rc2 in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git *** WARNING: tag release-1.5.0-rc2 was deleted! *** tag was 994792a5 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository.
[flink-kubernetes-operator] annotated tag release-1.5.0-rc1 deleted (was 6dd8234b)
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a change to annotated tag release-1.5.0-rc1 in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git *** WARNING: tag release-1.5.0-rc1 was deleted! *** tag was 6dd8234b This change permanently discards the following revisions: discard 6f088940 [release] Update version to 1.5.0
[flink-kubernetes-operator] annotated tag release-1.5.0 updated (be07be7d -> 68db8969)
This is an automated email from the ASF dual-hosted git repository. gyfora pushed a change to annotated tag release-1.5.0 in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git *** WARNING: tag release-1.5.0 was modified! *** from be07be7d (commit) to 68db8969 (tag) tagging be07be7dfee037dec6390d91b24f3ff2f2664e04 (commit) by Gyula Fora on Wed May 17 10:13:58 2023 +0200 - Log - Release Kubernetes Operator 1.5.0 -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEC0o0rd/6K7VOtyCyIfBjA7h9r/EFAmRkjMYACgkQIfBjA7h9 r/HbAg/9FOA5uk1o0agx3Uh1rVQarr0SAGavdfDV3KYIPf/oybrOm+nantpdJWYf FEeko43KgGeN6J9hKaiintNXPD5aMW/0A/ClRQTInnY/+kB0ZI/qkBKco0o9EHBJ TF+kNw3OwBi97VTO5il7xqPIgy7dX5WMylF7VX5PRLfsK35j0DPDKBjWryKtIRn2 7MiWSBzYU6g/e3LMlhBUFLHOB+ct9ZiRdT8nIMuiGhUSshQ3CqBXXjL9zMTbvV2T vTZP1H0CfF2SK+bfRB8Ye8wuS9jn0UYrbEjDQUj7mijFyquDEsD3SfiVGX9AwNIS 91j6KwccwgNvXQaMnfBlX7QhSOO3ryP9RtA8uRtiXrGWMlHrRam7quDx5ya82YQk VY32hUCiIz1PWxVFRG5MPdZHlr0sdevmTSWVi1XNgZxFy4QLdWb0/EtKVTsKA0o7 Fmq/PHssPzcZsev3XbaT47yhfFgBsUSGrGQ1VjnqfPLfQpH8rAT81utmOjtCPlZE PszX0yXG+rAwmMSZYkzm3TqsM9iGwt3ZMJ0XHJOulfCg2MuD1QANLr5Ai3ZctdSj 5Yk4AbsyslMMU6d0wRCeTa7ClRqfaVLeyonSXf8TtyxNe43fSOHkvpIyHDh4venb zDzvkLvq36V7AUio68+/eOSiRR9AWTTpFxDMmSn1IeEeq7mxjOQ= =88wS -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
svn commit: r61933 - /release/flink/flink-kubernetes-operator-1.3.1/
Author: gyfora Date: Wed May 17 08:12:14 2023 New Revision: 61933 Log: Remove old Kubernetes Operator release Removed: release/flink/flink-kubernetes-operator-1.3.1/
svn commit: r61932 - /dev/flink/flink-kubernetes-operator-1.5.0-rc2/ /release/flink/flink-kubernetes-operator-1.5.0/
Author: gyfora Date: Wed May 17 08:08:57 2023 New Revision: 61932 Log: Release Apache Flink Kubernetes Operator 1.5.0 Added: release/flink/flink-kubernetes-operator-1.5.0/ - copied from r61931, dev/flink/flink-kubernetes-operator-1.5.0-rc2/ Removed: dev/flink/flink-kubernetes-operator-1.5.0-rc2/