[GitHub] [flink] twalthr closed pull request #14309: [FLINK-20485][table] Improve performance of data views
twalthr closed pull request #14309: URL: https://github.com/apache/flink/pull/14309 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14328: [FLINK-19659][table-planner] Array type supports equals and not_equal…
flinkbot commented on pull request #14328: URL: https://github.com/apache/flink/pull/14328#issuecomment-740445577 ## CI report: * b4a79b01e25ccc9ef50019bcae53da401f36041e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14311: [FLINK-20490][table-planner-blink] Don't share inputs between FlinkPhysicalRel and ExecNode
flinkbot edited a comment on pull request #14311: URL: https://github.com/apache/flink/pull/14311#issuecomment-738797093 ## CI report: * 8097824271a1688deaaf384d42b46ff1b16ca52f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10527) * cfbc435a4778a36431108410ef22698c293cd132 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19659) Array type supports equals and not_equals operator when element types are different but castable
[
https://issues.apache.org/jira/browse/FLINK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hailong wang updated FLINK-19659:
-
Description:
Currently, Array type supports `equals` and `not_equals` when element types are
the same or can not be cased. For example,
{code:java}
Array[1] <> Array[1] -> false{code}
{code:java}
Array[1] <> Array[cast(1 as bigint)] -> false
{code}
But for the element types which are castable, it will throw error,
{code:java}
org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast from
'ARRAY NOT NULL' to 'ARRAY NOT
NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
from 'ARRAY NOT NULL' to 'ARRAY NOT NULL'. at
org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
But the result should be false or true, for example,
{code:java}
/Array[1] <> Array[cast(1 as bigint)] -> false
{code}
BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
other issues to track those.
was:
Currently, Array type supports `equals` and `not_equals` when element types are
the same or can not be cased. For example,
{code:java}
Array[1] <> Array[1] -> true{code}
{code:java}
Array[1] <> Array[cast(1 as bigint)] -> false
{code}
But for the element types which are castable, it will throw error,
{code:java}
org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast from
'ARRAY NOT NULL' to 'ARRAY NOT
NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
from 'ARRAY NOT NULL' to 'ARRAY NOT NULL'. at
org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
But the result should be false or true, for example,
{code:java}
/Array[1] <> Array[cast(1 as bigint)] -> false
{code}
BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
other issues to track those.
> Array type supports equals and not_equals operator when element types are
> different but castable
>
>
> Key: FLINK-19659
> URL: https://issues.apache.org/jira/browse/FLINK-19659
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Assignee: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, Array type supports `equals` and `not_equals` when element types
> are the same or can not be cased. For example,
> {code:java}
> Array[1] <> Array[1] -> false{code}
> {code:java}
> Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
> But for the element types which are castable, it will throw error,
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
> from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported
> cast from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'. at
> org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
> But the result should be false or true, for example,
> {code:java}
> /Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
>
> BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
> other issues to track those.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19659) Array type supports equals and not_equals operator when element types are different but castable
[
https://issues.apache.org/jira/browse/FLINK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hailong wang updated FLINK-19659:
-
Description:
Currently, Array type supports `equals` and `not_equals` when element types are
the same or can not be cased. For example,
{code:java}
Array[1] <> Array[1] -> true{code}
{code:java}
Array[1] <> Array[cast(1 as bigint)] -> false
{code}
But for the element types which are castable, it will throw error,
{code:java}
org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast from
'ARRAY NOT NULL' to 'ARRAY NOT
NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
from 'ARRAY NOT NULL' to 'ARRAY NOT NULL'. at
org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
But the result should be false or true, for example,
{code:java}
/Array[1] <> Array[cast(1 as bigint)] -> false
{code}
BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
other issues to track those.
was:
Currently, Array type supports `equals` and `not_equals` when element types are
the same or can not be cased. For example,
{code:java}
Array[1] <> Array[1] -> true{code}
{code:java}
Array[1] <> Array[cast(x'01' as binary)] -> false
{code}
But for the element types which are castable, it will throw error,
{code:java}
org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast from
'ARRAY NOT NULL' to 'ARRAY NOT
NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
from 'ARRAY NOT NULL' to 'ARRAY NOT NULL'. at
org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
But the result should be false or true, for example,
{code:java}
/Array[1] <> Array[cast(1 as bigint)] -> true
{code}
BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
other issues to track those.
> Array type supports equals and not_equals operator when element types are
> different but castable
>
>
> Key: FLINK-19659
> URL: https://issues.apache.org/jira/browse/FLINK-19659
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Assignee: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, Array type supports `equals` and `not_equals` when element types
> are the same or can not be cased. For example,
> {code:java}
> Array[1] <> Array[1] -> true{code}
> {code:java}
> Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
> But for the element types which are castable, it will throw error,
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
> from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported
> cast from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'. at
> org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
> But the result should be false or true, for example,
> {code:java}
> /Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
>
> BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
> other issues to track those.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14328: [FLINK-19659][table-planner] Array type supports equals and not_equal…
flinkbot commented on pull request #14328: URL: https://github.com/apache/flink/pull/14328#issuecomment-740441995 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b4a79b01e25ccc9ef50019bcae53da401f36041e (Tue Dec 08 07:38:24 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] HuangXingBo commented on a change in pull request #397: Add 1.12 Release announcement.
HuangXingBo commented on a change in pull request #397:
URL: https://github.com/apache/flink-web/pull/397#discussion_r538097233
##
File path: _posts/2020-12-04-release-1.12.0.md
##
@@ -0,0 +1,332 @@
+---
+layout: post
+title: "Apache Flink 1.12.0 Release Announcement"
+date: 2020-12-04T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+- aljoscha:
+ name: "Aljoscha Krettek"
+ twitter: "aljoscha"
+
+excerpt: The Apache Flink community is excited to announce the release of
Flink 1.12.0! Close to 300 contributors worked on over 1k tickets to bring
significant improvements to usability as well as new features to Flink users
across the whole API stack. We're particularly excited about adding efficient
batch execution to the DataStream API, Kubernetes HA as an alternative to
ZooKeeper, support for upsert mode in the Kafka SQL connector and the new
Python DataStream API! Read on for all major new features and improvements,
important changes to be aware of and what to expect moving forward!
+---
+
+The Apache Flink community is excited to announce the release of Flink 1.12.0!
Close to 300 contributors worked on over 1k tickets to bring significant
improvements to usability as well as new features that simplify (and unify)
Flink handling across the API stack.
+
+**Release Highlights**
+
+* The community has added support for **efficient batch execution** in the
DataStream API. This is the next major milestone towards achieving a truly
unified runtime for both batch and stream processing.
+
+* **Kubernetes-based High Availability (HA)** was implemented as an
alternative to ZooKeeper for highly available production setups.
+
+* The Kafka SQL connector has been extended to work in **upsert mode**,
supported by the ability to handle **connector metadata** in SQL DDL.
**Temporal table joins** can now also be fully expressed in SQL, no longer
depending on the Table API.
+
+* Support for the **DataStream API in PyFlink** expands its usage to more
complex scenarios that require fine-grained control over state and time, and
it’s now possible to deploy PyFlink jobs natively on **Kubernetes**.
+
+This blog post describes all major new features and improvements, important
changes to be aware of and what to expect moving forward.
+
+{% toc %}
+
+The binary distribution and source artifacts are now available on the updated
[Downloads page]({{ site.baseurl }}/downloads.html) of the Flink website, and
the most recent distribution of PyFlink is available on
[PyPI](https://pypi.org/project/apache-flink/). Please review the [release
notes]({{ site.DOCS_BASE_URL
}}flink-docs-release-1.12/release-notes/flink-1.12.html) carefully, and check
the complete [release
changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12348263&styleName=Html&projectId=12315522)
and [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.12/)
for more details.
+
+We encourage you to download the release and share your feedback with the
community through the [Flink mailing
lists](https://flink.apache.org/community.html#mailing-lists) or
[JIRA](https://issues.apache.org/jira/projects/FLINK/summary).
+
+## New Features and Improvements
+
+### Batch Execution Mode in the DataStream API
+
+Flink’s core APIs have developed organically over the lifetime of the project,
and were initially designed with specific use cases in mind. And while the
Table API/SQL already has unified operators, using lower-level abstractions
still requires you to choose between two semantically different APIs for batch
(DataSet API) and streaming (DataStream API). Since _a batch is a subset of an
unbounded stream_, there are some clear advantages to consolidating them under
a single API:
+
+* **Reusability:** efficient batch and stream processing under the same API
would allow you to easily switch between both execution modes without rewriting
any code. So, a job could be easily reused to process real-time and historical
data.
+
+* **Operational simplicity:** providing a unified API would mean using a
single set of connectors, maintaining a single codebase and being able to
easily implement mixed execution pipelines _e.g._ for use cases like
backfilling.
+
+With these advantages in mind, the community has taken the first step towards
the unification of the DataStream API: supporting efficient batch execution
([FLIP-134](https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API)).
This means that, in the long run, the DataSet API will be deprecated and
subsumed by the DataStream API and the Table API/SQL
([FLIP-131](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741)).
+
+**Batch for Bounded Streams**
+
+You could already use the DataStream API to process bounded streams (_e.g._
files), with the limitation that the runtime is not “aware” that the job is
bounded. To
[jira] [Updated] (FLINK-19659) Array type supports equals and not_equals operator when element types are different but castable
[
https://issues.apache.org/jira/browse/FLINK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19659:
---
Labels: pull-request-available (was: )
> Array type supports equals and not_equals operator when element types are
> different but castable
>
>
> Key: FLINK-19659
> URL: https://issues.apache.org/jira/browse/FLINK-19659
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Assignee: hailong wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently, Array type supports `equals` and `not_equals` when element types
> are the same or can not be cased. For example,
> {code:java}
> Array[1] <> Array[1] -> true{code}
> {code:java}
> Array[1] <> Array[cast(x'01' as binary)] -> false
> {code}
> But for the element types which are castable, it will throw error,
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
> from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported
> cast from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'. at
> org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
> But the result should be false or true, for example,
> {code:java}
> /Array[1] <> Array[cast(1 as bigint)] -> true
> {code}
>
> BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
> other issues to track those.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangxlong opened a new pull request #14328: [FLINK-19659][table-planner] Array type supports equals and not_equal…
wangxlong opened a new pull request #14328: URL: https://github.com/apache/flink/pull/14328 …s operator when element types are different but castable ## What is the purpose of the change Array type supports equals and not_equals operator when element types are different but castable ## Brief change log Add the cast code between array and array ## Verifying this change This change added tests and can be verified as follows: 1. Add test to ScalarOperatorsTest#testCompareOperator to verify the comparison between array and array 2. Add test to ScalarOperatorsTest#testCast to verify that it is castable between array and array ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no / don't know) - The runtime per-record code paths (performance sensitive): (no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no / don't know) - The S3 file system connector: (no / don't know) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] HuangXingBo commented on a change in pull request #397: Add 1.12 Release announcement.
HuangXingBo commented on a change in pull request #397:
URL: https://github.com/apache/flink-web/pull/397#discussion_r538097233
##
File path: _posts/2020-12-04-release-1.12.0.md
##
@@ -0,0 +1,332 @@
+---
+layout: post
+title: "Apache Flink 1.12.0 Release Announcement"
+date: 2020-12-04T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+- aljoscha:
+ name: "Aljoscha Krettek"
+ twitter: "aljoscha"
+
+excerpt: The Apache Flink community is excited to announce the release of
Flink 1.12.0! Close to 300 contributors worked on over 1k tickets to bring
significant improvements to usability as well as new features to Flink users
across the whole API stack. We're particularly excited about adding efficient
batch execution to the DataStream API, Kubernetes HA as an alternative to
ZooKeeper, support for upsert mode in the Kafka SQL connector and the new
Python DataStream API! Read on for all major new features and improvements,
important changes to be aware of and what to expect moving forward!
+---
+
+The Apache Flink community is excited to announce the release of Flink 1.12.0!
Close to 300 contributors worked on over 1k tickets to bring significant
improvements to usability as well as new features that simplify (and unify)
Flink handling across the API stack.
+
+**Release Highlights**
+
+* The community has added support for **efficient batch execution** in the
DataStream API. This is the next major milestone towards achieving a truly
unified runtime for both batch and stream processing.
+
+* **Kubernetes-based High Availability (HA)** was implemented as an
alternative to ZooKeeper for highly available production setups.
+
+* The Kafka SQL connector has been extended to work in **upsert mode**,
supported by the ability to handle **connector metadata** in SQL DDL.
**Temporal table joins** can now also be fully expressed in SQL, no longer
depending on the Table API.
+
+* Support for the **DataStream API in PyFlink** expands its usage to more
complex scenarios that require fine-grained control over state and time, and
it’s now possible to deploy PyFlink jobs natively on **Kubernetes**.
+
+This blog post describes all major new features and improvements, important
changes to be aware of and what to expect moving forward.
+
+{% toc %}
+
+The binary distribution and source artifacts are now available on the updated
[Downloads page]({{ site.baseurl }}/downloads.html) of the Flink website, and
the most recent distribution of PyFlink is available on
[PyPI](https://pypi.org/project/apache-flink/). Please review the [release
notes]({{ site.DOCS_BASE_URL
}}flink-docs-release-1.12/release-notes/flink-1.12.html) carefully, and check
the complete [release
changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12348263&styleName=Html&projectId=12315522)
and [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.12/)
for more details.
+
+We encourage you to download the release and share your feedback with the
community through the [Flink mailing
lists](https://flink.apache.org/community.html#mailing-lists) or
[JIRA](https://issues.apache.org/jira/projects/FLINK/summary).
+
+## New Features and Improvements
+
+### Batch Execution Mode in the DataStream API
+
+Flink’s core APIs have developed organically over the lifetime of the project,
and were initially designed with specific use cases in mind. And while the
Table API/SQL already has unified operators, using lower-level abstractions
still requires you to choose between two semantically different APIs for batch
(DataSet API) and streaming (DataStream API). Since _a batch is a subset of an
unbounded stream_, there are some clear advantages to consolidating them under
a single API:
+
+* **Reusability:** efficient batch and stream processing under the same API
would allow you to easily switch between both execution modes without rewriting
any code. So, a job could be easily reused to process real-time and historical
data.
+
+* **Operational simplicity:** providing a unified API would mean using a
single set of connectors, maintaining a single codebase and being able to
easily implement mixed execution pipelines _e.g._ for use cases like
backfilling.
+
+With these advantages in mind, the community has taken the first step towards
the unification of the DataStream API: supporting efficient batch execution
([FLIP-134](https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API)).
This means that, in the long run, the DataSet API will be deprecated and
subsumed by the DataStream API and the Table API/SQL
([FLIP-131](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741)).
+
+**Batch for Bounded Streams**
+
+You could already use the DataStream API to process bounded streams (_e.g._
files), with the limitation that the runtime is not “aware” that the job is
bounded. To
[GitHub] [flink-web] zhuzhurk commented on a change in pull request #397: Add 1.12 Release announcement.
zhuzhurk commented on a change in pull request #397:
URL: https://github.com/apache/flink-web/pull/397#discussion_r538090393
##
File path: _posts/2020-12-04-release-1.12.0.md
##
@@ -0,0 +1,332 @@
+---
+layout: post
+title: "Apache Flink 1.12.0 Release Announcement"
+date: 2020-12-04T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+- aljoscha:
+ name: "Aljoscha Krettek"
+ twitter: "aljoscha"
+
+excerpt: The Apache Flink community is excited to announce the release of
Flink 1.12.0! Close to 300 contributors worked on over 1k tickets to bring
significant improvements to usability as well as new features to Flink users
across the whole API stack. We're particularly excited about adding efficient
batch execution to the DataStream API, Kubernetes HA as an alternative to
ZooKeeper, support for upsert mode in the Kafka SQL connector and the new
Python DataStream API! Read on for all major new features and improvements,
important changes to be aware of and what to expect moving forward!
+---
+
+The Apache Flink community is excited to announce the release of Flink 1.12.0!
Close to 300 contributors worked on over 1k tickets to bring significant
improvements to usability as well as new features that simplify (and unify)
Flink handling across the API stack.
+
+**Release Highlights**
+
+* The community has added support for **efficient batch execution** in the
DataStream API. This is the next major milestone towards achieving a truly
unified runtime for both batch and stream processing.
+
+* **Kubernetes-based High Availability (HA)** was implemented as an
alternative to ZooKeeper for highly available production setups.
+
+* The Kafka SQL connector has been extended to work in **upsert mode**,
supported by the ability to handle **connector metadata** in SQL DDL.
**Temporal table joins** can now also be fully expressed in SQL, no longer
depending on the Table API.
+
+* Support for the **DataStream API in PyFlink** expands its usage to more
complex scenarios that require fine-grained control over state and time, and
it’s now possible to deploy PyFlink jobs natively on **Kubernetes**.
+
+This blog post describes all major new features and improvements, important
changes to be aware of and what to expect moving forward.
+
+{% toc %}
+
+The binary distribution and source artifacts are now available on the updated
[Downloads page]({{ site.baseurl }}/downloads.html) of the Flink website, and
the most recent distribution of PyFlink is available on
[PyPI](https://pypi.org/project/apache-flink/). Please review the [release
notes]({{ site.DOCS_BASE_URL
}}flink-docs-release-1.12/release-notes/flink-1.12.html) carefully, and check
the complete [release
changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12348263&styleName=Html&projectId=12315522)
and [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.12/)
for more details.
+
+We encourage you to download the release and share your feedback with the
community through the [Flink mailing
lists](https://flink.apache.org/community.html#mailing-lists) or
[JIRA](https://issues.apache.org/jira/projects/FLINK/summary).
+
+## New Features and Improvements
+
+### Batch Execution Mode in the DataStream API
+
+Flink’s core APIs have developed organically over the lifetime of the project,
and were initially designed with specific use cases in mind. And while the
Table API/SQL already has unified operators, using lower-level abstractions
still requires you to choose between two semantically different APIs for batch
(DataSet API) and streaming (DataStream API). Since _a batch is a subset of an
unbounded stream_, there are some clear advantages to consolidating them under
a single API:
+
+* **Reusability:** efficient batch and stream processing under the same API
would allow you to easily switch between both execution modes without rewriting
any code. So, a job could be easily reused to process real-time and historical
data.
+
+* **Operational simplicity:** providing a unified API would mean using a
single set of connectors, maintaining a single codebase and being able to
easily implement mixed execution pipelines _e.g._ for use cases like
backfilling.
+
+With these advantages in mind, the community has taken the first step towards
the unification of the DataStream API: supporting efficient batch execution
([FLIP-134](https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API)).
This means that, in the long run, the DataSet API will be deprecated and
subsumed by the DataStream API and the Table API/SQL
([FLIP-131](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741)).
+
+**Batch for Bounded Streams**
+
+You could already use the DataStream API to process bounded streams (_e.g._
files), with the limitation that the runtime is not “aware” that the job is
bounded. To opt
[jira] [Closed] (FLINK-20300) Create Flink 1.12 release notes
[ https://issues.apache.org/jira/browse/FLINK-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger closed FLINK-20300. -- Fix Version/s: 1.12.1 Resolution: Fixed > Create Flink 1.12 release notes > --- > > Key: FLINK-20300 > URL: https://issues.apache.org/jira/browse/FLINK-20300 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Critical > Labels: pull-request-available > Fix For: 1.13.0, 1.12.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20300) Create Flink 1.12 release notes
[ https://issues.apache.org/jira/browse/FLINK-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245700#comment-17245700 ] Robert Metzger commented on FLINK-20300: merged to master in https://github.com/apache/flink/commit/d94c7a451d22f861bd3f79435f777b427020eba0 merged to release-1.12 in https://github.com/apache/flink/commit/3c83f315774805321639358f9703bfd3e96f82ff > Create Flink 1.12 release notes > --- > > Key: FLINK-20300 > URL: https://issues.apache.org/jira/browse/FLINK-20300 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Critical > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rmetzger closed pull request #14195: [FLINK-20300] Add Flink 1.12 release notes
rmetzger closed pull request #14195: URL: https://github.com/apache/flink/pull/14195 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16027) kafka connector's 'connector.topic' property should be optional rather than required
[ https://issues.apache.org/jira/browse/FLINK-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-16027: --- Fix Version/s: (was: 1.12.0) 1.13.0 > kafka connector's 'connector.topic' property should be optional rather than > required > > > Key: FLINK-16027 > URL: https://issues.apache.org/jira/browse/FLINK-16027 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Bowen Li >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18159) Add support for trimming stack traces in user-facing components
[
https://issues.apache.org/jira/browse/FLINK-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18159:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Add support for trimming stack traces in user-facing components
> ---
>
> Key: FLINK-18159
> URL: https://issues.apache.org/jira/browse/FLINK-18159
> Project: Flink
> Issue Type: New Feature
> Components: Command Line Client, Runtime / REST
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.13.0
>
>
> Add a verbosity query parameter / flag(==config option) into the REST API /
> CLI to trim the exception stack traces, which means excluding the location
> information.
> The result would be something like this:
> {code}
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
> Caused by: java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobExecutionException: Could not set up
> JobManager
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not
> set up JobManager
> Caused by: java.io.FileNotFoundException: Cannot find checkpoint or savepoint
> file/directory 'ashudasd' on file system 'file'.
> {code}
> This approach renders even the biggest stack traces fairly readable, and is
> rather convenient since it only requires changes in the actual user-facing
> components.
> Logging would not be impacted by this.
> The trimming was already implemented in this
> [PR|https://github.com/apache/flink/pull/12392], but the flags are missing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15801) Timestamp extractor created from properties does not work for some physical fields
[
https://issues.apache.org/jira/browse/FLINK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15801:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Timestamp extractor created from properties does not work for some physical
> fields
> --
>
> Key: FLINK-15801
> URL: https://issues.apache.org/jira/browse/FLINK-15801
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.3, 1.13.0, 1.11.4
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a timestamp extractor is created from properties it can not use a physical
> field if the name of that field is equal to the logical field of the rowtime
> field.
> The code below fails:
> {code}
> StreamExecutionEnvironment streamExecutionEnvironment =
> StreamExecutionEnvironment.getExecutionEnvironment();
>
> streamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
> StreamTableEnvironment fsTableEnv =
> StreamTableEnvironment.create(streamExecutionEnvironment);
> String allEventsTable = "allEventsTable";
> fsTableEnv.connect(new Kafka()
> .version("universal")
> .topic("events")
> .property("zookeeper.connect", "")
> .property("bootstrap.servers", "localhost:9092")
> .property("group.id", "dummyquery").startFromLatest())
> .withSchema(new Schema()
> .field("rule_id", Types.INT)
> .field("sourceAddress", Types.STRING)
> .field("deviceProduct", Types.STRING)
> .field("destHost", Types.STRING)
> .field("extra", Types.STRING)
> .field("rowtime", Types.SQL_TIMESTAMP)
> .rowtime(new
> Rowtime().timestampsFromField("rowtime").watermarksPeriodicBounded(2000))
> )
> .withFormat(new
> Json().failOnMissingField(false).deriveSchema())
> .inAppendMode()
> .registerTableSource(allEventsTable);
> Table result = fsTableEnv.sqlQuery("select * from
> allEventsTable where sourceAddress='12345431'");
> DataStream alert = fsTableEnv.toAppendStream(result, Row.class);
> alert.print();
> {code}
> with exception:
> {code}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Field 'rowtime' could not be resolved by the field mapping.
> at
> org.apache.flink.table.sources.TableSourceValidation.resolveField(TableSourceValidation.java:245)
> at
> org.apache.flink.table.sources.TableSourceValidation.lambda$validateTimestampExtractorArguments$6(TableSourceValidation.java:202)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
> {code}
> The problem is that the field is removed from the {{FieldMapping}} in
> {{org.apache.flink.table.descriptors.SchemaValidator#deriveFieldMapping}}.
> One possible solution could be to add:
> {code}
> if (isRowtime) {
> Optional timestampSource =
> properties.getOptionalString(SCHEMA + "." + i + "." +
> ROWTIME_TIMESTAMPS_FROM);
> timestampSource.ifPresent(s ->
> mapping.put(s, s));
> }
> {code}
> We should also consider the case what happens if we compute generated columns
> on fields that were pruned in a similar way.
> Reported by a user:
> https://stackoverflow.com/questions/59857057/how-to-define-an-apache-flink-table-with-row-time-attribute
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14510) Remove the lazy vertex attaching mechanism from ExecutionGraph
[ https://issues.apache.org/jira/browse/FLINK-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14510: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Remove the lazy vertex attaching mechanism from ExecutionGraph > -- > > Key: FLINK-14510 > URL: https://issues.apache.org/jira/browse/FLINK-14510 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Fix For: 1.13.0 > > > Currently in production, the vertex attaching is only invoked right after the > ExecutionGraph is created in ExecutionGraphBuilder. That means lazy attaching > is not necessary at the moment. It however adds extra complexity to > ExecutionGraph, since we need to assume that the vertices may be not > initialized or even get changed. > Moreover, attaching vertices after a job starts scheduling is an undefined > behavior which would not work properly. > I'd propose to remove the lazy attaching mechanism, and do vertices building > and related components initialization in ExecutionGraph constructor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-19698:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.3, 1.13.0
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rmetzger commented on pull request #14195: [FLINK-20300] Add Flink 1.12 release notes
rmetzger commented on pull request #14195: URL: https://github.com/apache/flink/pull/14195#issuecomment-740422715 I addressed the remaining comments. I'm now merging the release notes! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14195: [FLINK-20300] Add Flink 1.12 release notes
rmetzger commented on a change in pull request #14195:
URL: https://github.com/apache/flink/pull/14195#discussion_r538081905
##
File path: docs/release-notes/flink-1.12.md
##
@@ -0,0 +1,173 @@
+---
+title: "Release Notes - Flink 1.12"
+---
+
+
+
+These release notes discuss important aspects, such as configuration, behavior,
+or dependencies, that changed between Flink 1.11 and Flink 1.12. Please read
+these notes carefully if you are planning to upgrade your Flink version to
1.12.
+
+* This will be replaced by the TOC
+{:toc}
+
+
+### APIs
+
+ Remove deprecated methods in ExecutionConfig
[FLINK-19084](https://issues.apache.org/jira/browse/FLINK-19084)
+
+Deprecated method `ExecutionConfig#isLatencyTrackingEnabled` was removed, you
can use `ExecutionConfig#getLatencyTrackingInterval` instead.
+
+Deprecated and methods without effect were removed:
`ExecutionConfig#enable/disableSysoutLogging`,
`ExecutionConfig#set/isFailTaskOnCheckpointError`.
+
+Removed `-q` flag from cli. The option had no effect.
+
+ Remove deprecated RuntimeContext#getAllAccumulators
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
+
+The deprecated method `RuntimeContext#getAllAccumulators` was removed. Please
use `RuntimeContext#getAccumulator` instead.
+
+ Deprecated CheckpointConfig#setPreferCheckpointForRecovery due to risk of
data loss [FLINK-20441](https://issues.apache.org/jira/browse/FLINK-20441)
+
+The `CheckpointConfig#setPreferCheckpointForRecovery` method has been
deprecated, because preferring older checkpoints over newer savepoints for
recovery can lead to data loss.
+
+ FLIP-134: Batch execution for the DataStream API
+
+- Allow explicitly configuring time behaviour on `KeyedStream.intervalJoin()`
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
+
+ Before Flink 1.12 the `KeyedStream.intervalJoin()` operation was changing
behavior based on the globally set Stream TimeCharacteristic. In Flink 1.12 we
introduced explicit `inProcessingTime()` and `inEventTime()` methods on
`IntervalJoin` and the join no longer changes behaviour based on the global
characteristic.
+
+- Deprecate `timeWindow()` operations in DataStream API
[FLINK-19318](https://issues.apache.org/jira/browse/FLINK-19318)
+
+ In Flink 1.12 we deprecated the `timeWindow()` operations in the DataStream
API. Please use `window(WindowAssigner)` with either a
`TumblingEventTimeWindows`, `SlidingEventTimeWindows`,
`TumblingProcessingTimeWindows`, or `SlidingProcessingTimeWindows`. For more
information, see the deprecation description of
`TimeCharacteristic`/`setStreamTimeCharacteristic`.
+
+- Deprecate `StreamExecutionEnvironment.setStreamTimeCharacteristic()` and
`TimeCharacteristic`
[FLINK-19319](https://issues.apache.org/jira/browse/FLINK-19319)
+
+ In Flink 1.12 the default stream time characteristic has been changed to
`EventTime`, thus you don't need to call this method for enabling event-time
support anymore. Explicitly using processing-time windows and timers works in
event-time mode. If you need to disable watermarks, please use
`ExecutionConfig.setAutoWatermarkInterval(long)`. If you are using
`IngestionTime`, please manually set an appropriate `WatermarkStrategy`. If you
are using generic "time window" operations (for example
`KeyedStream.timeWindow()` that change behaviour based on the time
characteristic, please use equivalent operations that explicitly specify
processing time or event time).
+
+- Allow explicitly configuring time behaviour on CEP PatternStream
[FLINK-19326](https://issues.apache.org/jira/browse/FLINK-19326)
+
+ Before Flink 1.12 the CEP operations were changing their behavior based on
the globally set Stream TimeCharacteristic. In Flink 1.12 we introduced
explicit `inProcessingTime()` and `inEventTime()` methods on `PatternStream`
and the CEP operations no longer change their behaviour based on the global
characteristic.
+
+ API cleanups
+
+- Remove remaining UdfAnalyzer configurations
[FLINK-13857](https://issues.apache.org/jira/browse/FLINK-13857)
+
+ The `ExecutionConfig#get/setCodeAnalysisMode` method and `SkipCodeAnalysis`
class were removed. They took no effect even before that change, therefore
there is no need to use any of these.
+
+- Remove deprecated `DataStream#split`
[FLINK-19083](https://issues.apache.org/jira/browse/FLINK-19083)
+
+ The `DataStream#split()` operation has been removed after being marked as
deprecated for a couple of versions. Please use [Side Outputs]({% link
dev/stream/side_output.md %})) instead.
+
+- Remove deprecated `DataStream#fold()` method and all related classes
[FLINK-19035](https://issues.apache.org/jira/browse/FLINK-19035)
+
+ The long deprecated `(Windowed)DataStream#fold` was removed in 1.12. Please
use other operations such as e.g. `(Windowed)DataStream#reduce` that perform
better in distributed systems.
+
+ Extend CompositeTypeSerializerSnapshot to allow
[GitHub] [flink] rmetzger commented on a change in pull request #14195: [FLINK-20300] Add Flink 1.12 release notes
rmetzger commented on a change in pull request #14195:
URL: https://github.com/apache/flink/pull/14195#discussion_r538081301
##
File path: docs/release-notes/flink-1.12.md
##
@@ -0,0 +1,173 @@
+---
+title: "Release Notes - Flink 1.12"
+---
+
+
+
+These release notes discuss important aspects, such as configuration, behavior,
+or dependencies, that changed between Flink 1.11 and Flink 1.12. Please read
+these notes carefully if you are planning to upgrade your Flink version to
1.12.
+
+* This will be replaced by the TOC
+{:toc}
+
+
+### APIs
+
+ Remove deprecated methods in ExecutionConfig
[FLINK-19084](https://issues.apache.org/jira/browse/FLINK-19084)
+
+Deprecated method `ExecutionConfig#isLatencyTrackingEnabled` was removed, you
can use `ExecutionConfig#getLatencyTrackingInterval` instead.
+
+Deprecated and methods without effect were removed:
`ExecutionConfig#enable/disableSysoutLogging`,
`ExecutionConfig#set/isFailTaskOnCheckpointError`.
+
+Removed `-q` flag from cli. The option had no effect.
+
+ Remove deprecated RuntimeContext#getAllAccumulators
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
+
+The deprecated method `RuntimeContext#getAllAccumulators` was removed. Please
use `RuntimeContext#getAccumulator` instead.
+
+ Deprecated CheckpointConfig#setPreferCheckpointForRecovery due to risk of
data loss [FLINK-20441](https://issues.apache.org/jira/browse/FLINK-20441)
+
+The `CheckpointConfig#setPreferCheckpointForRecovery` method has been
deprecated, because preferring older checkpoints over newer savepoints for
recovery can lead to data loss.
+
+ FLIP-134: Batch execution for the DataStream API
+
+- Allow explicitly configuring time behaviour on `KeyedStream.intervalJoin()`
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
Review comment:
Very good catch! Thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14195: [FLINK-20300] Add Flink 1.12 release notes
rmetzger commented on a change in pull request #14195:
URL: https://github.com/apache/flink/pull/14195#discussion_r538080702
##
File path: docs/release-notes/flink-1.12.md
##
@@ -0,0 +1,173 @@
+---
+title: "Release Notes - Flink 1.12"
+---
+
+
+
+These release notes discuss important aspects, such as configuration, behavior,
+or dependencies, that changed between Flink 1.11 and Flink 1.12. Please read
+these notes carefully if you are planning to upgrade your Flink version to
1.12.
+
+* This will be replaced by the TOC
+{:toc}
+
+
+### APIs
+
+ Remove deprecated methods in ExecutionConfig
[FLINK-19084](https://issues.apache.org/jira/browse/FLINK-19084)
+
+Deprecated method `ExecutionConfig#isLatencyTrackingEnabled` was removed, you
can use `ExecutionConfig#getLatencyTrackingInterval` instead.
+
+Deprecated and methods without effect were removed:
`ExecutionConfig#enable/disableSysoutLogging`,
`ExecutionConfig#set/isFailTaskOnCheckpointError`.
+
+Removed `-q` flag from cli. The option had no effect.
+
+ Remove deprecated RuntimeContext#getAllAccumulators
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
+
+The deprecated method `RuntimeContext#getAllAccumulators` was removed. Please
use `RuntimeContext#getAccumulator` instead.
+
+ Deprecated CheckpointConfig#setPreferCheckpointForRecovery due to risk of
data loss [FLINK-20441](https://issues.apache.org/jira/browse/FLINK-20441)
+
+The `CheckpointConfig#setPreferCheckpointForRecovery` method has been
deprecated, because preferring older checkpoints over newer savepoints for
recovery can lead to data loss.
+
+ FLIP-134: Batch execution for the DataStream API
+
+- Allow explicitly configuring time behaviour on `KeyedStream.intervalJoin()`
[FLINK-19032](https://issues.apache.org/jira/browse/FLINK-19032)
+
+ Before Flink 1.12 the `KeyedStream.intervalJoin()` operation was changing
behavior based on the globally set Stream TimeCharacteristic. In Flink 1.12 we
introduced explicit `inProcessingTime()` and `inEventTime()` methods on
`IntervalJoin` and the join no longer changes behaviour based on the global
characteristic.
+
+- Deprecate `timeWindow()` operations in DataStream API
[FLINK-19318](https://issues.apache.org/jira/browse/FLINK-19318)
+
+ In Flink 1.12 we deprecated the `timeWindow()` operations in the DataStream
API. Please use `window(WindowAssigner)` with either a
`TumblingEventTimeWindows`, `SlidingEventTimeWindows`,
`TumblingProcessingTimeWindows`, or `SlidingProcessingTimeWindows`. For more
information, see the deprecation description of
`TimeCharacteristic`/`setStreamTimeCharacteristic`.
Review comment:
I assume this has been added because the deprecation of the timeWindow()
is part of the time characteristic change, which is described in the
deprecation description.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-10292) Generate JobGraph in StandaloneJobClusterEntrypoint only once
[
https://issues.apache.org/jira/browse/FLINK-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-10292:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Generate JobGraph in StandaloneJobClusterEntrypoint only once
> -
>
> Key: FLINK-10292
> URL: https://issues.apache.org/jira/browse/FLINK-10292
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Till Rohrmann
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Currently the {{StandaloneJobClusterEntrypoint}} generates the {{JobGraph}}
> from the given user code every time it starts/is restarted. This can be
> problematic if the the {{JobGraph}} generation has side effects. Therefore,
> it would be better to generate the {{JobGraph}} only once and store it in HA
> storage instead from where to retrieve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-9407) Support orc rolling sink writer
[
https://issues.apache.org/jira/browse/FLINK-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-9407:
--
Fix Version/s: (was: 1.12.0)
1.13.0
> Support orc rolling sink writer
> ---
>
> Key: FLINK-9407
> URL: https://issues.apache.org/jira/browse/FLINK-9407
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem
>Reporter: zhangminglei
>Priority: Major
> Labels: pull-request-available, usability
> Fix For: 1.13.0
>
>
> Currently, we only support {{StringWriter}}, {{SequenceFileWriter}} and
> {{AvroKeyValueSinkWriter}}. I would suggest add an orc writer for rolling
> sink.
> Below, FYI.
> I tested the PR and verify the results with spark sql. Obviously, we can get
> the results of what we had written down before. But I will give more tests in
> the next couple of days. Including the performance under compression with
> short checkpoint intervals. And more UTs.
> {code:java}
> scala> spark.read.orc("hdfs://10.199.196.0:9000/data/hive/man/2018-07-06--21")
> res1: org.apache.spark.sql.DataFrame = [name: string, age: int ... 1 more
> field]
> scala>
> scala> res1.registerTempTable("tablerice")
> warning: there was one deprecation warning; re-run with -deprecation for
> details
> scala> spark.sql("select * from tablerice")
> res3: org.apache.spark.sql.DataFrame = [name: string, age: int ... 1 more
> field]
> scala> res3.show(3)
> +-+---+---+
> | name|age|married|
> +-+---+---+
> |Sagar| 26| false|
> |Sagar| 30| false|
> |Sagar| 34| false|
> +-+---+---+
> only showing top 3 rows
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11868) [filesystems] Introduce listStatusIterator API to file system
[
https://issues.apache.org/jira/browse/FLINK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-11868:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> [filesystems] Introduce listStatusIterator API to file system
> -
>
> Key: FLINK-11868
> URL: https://issues.apache.org/jira/browse/FLINK-11868
> Project: Flink
> Issue Type: New Feature
> Components: FileSystems
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Fix For: 1.13.0
>
>
> From existed experience, we know {{listStatus}} is expensive for many
> distributed file systems especially when the folder contains too many files.
> This method would not only block the thread until result is return but also
> could cause OOM due to the returned array of {{FileStatus}} is really large.
> I think we should already learn it from FLINK-7266 and FLINK-8540.
> However, list file status under a path is really helpful in many situations.
> Thankfully, many distributed file system noticed that and provide API such as
> {{[listStatusIterator|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatusIterator(org.apache.hadoop.fs.Path)]}}
> to call the file system on demand.
>
> We should also introduce this API and replace current implementation which
> used previous {{listStatus}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-10134) UTF-16 support for TextInputFormat
[
https://issues.apache.org/jira/browse/FLINK-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-10134:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> UTF-16 support for TextInputFormat
> --
>
> Key: FLINK-10134
> URL: https://issues.apache.org/jira/browse/FLINK-10134
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.4.2
>Reporter: David Dreyfus
>Assignee: Forward Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> It does not appear that Flink supports a charset encoding of "UTF-16". It
> particular, it doesn't appear that Flink consumes the Byte Order Mark (BOM)
> to establish whether a UTF-16 file is UTF-16LE or UTF-16BE.
>
> TextInputFormat.setCharset("UTF-16") calls DelimitedInputFormat.setCharset(),
> which sets TextInputFormat.charsetName and then modifies the previously set
> delimiterString to construct the proper byte string encoding of the the
> delimiter. This same charsetName is also used in TextInputFormat.readRecord()
> to interpret the bytes read from the file.
>
> There are two problems that this implementation would seem to have when using
> UTF-16.
> # delimiterString.getBytes(getCharset()) in DelimitedInputFormat.java will
> return a Big Endian byte sequence including the Byte Order Mark (BOM). The
> actual text file will not contain a BOM at each line ending, so the delimiter
> will never be read. Moreover, if the actual byte encoding of the file is
> Little Endian, the bytes will be interpreted incorrectly.
> # TextInputFormat.readRecord() will not see a BOM each time it decodes a
> byte sequence with the String(bytes, offset, numBytes, charset) call.
> Therefore, it will assume Big Endian, which may not always be correct. [1]
> [https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/io/TextInputFormat.java#L95]
>
> While there are likely many solutions, I would think that all of them would
> have to start by reading the BOM from the file when a Split is opened and
> then using that BOM to modify the specified encoding to a BOM specific one
> when the caller doesn't specify one, and to overwrite the caller's
> specification if the BOM is in conflict with the caller's specification. That
> is, if the BOM indicates Little Endian and the caller indicates UTF-16BE,
> Flink should rewrite the charsetName as UTF-16LE.
> I hope this makes sense and that I haven't been testing incorrectly or
> misreading the code.
>
> I've verified the problem on version 1.4.2. I believe the problem exists on
> all versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12193) Send TM "can be released status" with RM heartbeat
[ https://issues.apache.org/jira/browse/FLINK-12193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-12193: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Send TM "can be released status" with RM heartbeat > -- > > Key: FLINK-12193 > URL: https://issues.apache.org/jira/browse/FLINK-12193 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Andrey Zagrebin >Priority: Major > Labels: starter > Fix For: 1.13.0 > > > We introduced a conditional release of Task Executor in Resource Manager in > FLINK-10941. At the moment RM directly asks TE every release timeout whether > it can be released (all depending consumers are done). We can piggyback TE/RM > heartbeats for this purpose. In this case, we do not need additional RPC call > to TE gateway and could potentially release TE quicker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-11499) Extend StreamingFileSink BulkFormats to support arbitrary roll policies
[ https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-11499: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Extend StreamingFileSink BulkFormats to support arbitrary roll policies > --- > > Key: FLINK-11499 > URL: https://issues.apache.org/jira/browse/FLINK-11499 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Seth Wiesman >Priority: Major > Labels: usability > Fix For: 1.13.0 > > > Currently when using the StreamingFilleSink Bulk-encoding formats can only be > combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress > part file on every checkpoint. > However, many bulk formats such as parquet are most efficient when written as > large files; this is not possible when frequent checkpointing is enabled. > Currently the only work-around is to have long checkpoint intervals which is > not ideal. > > The StreamingFileSink should be enhanced to support arbitrary roll policy's > so users may write large bulk files while retaining frequent checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-9879) Find sane defaults for (advanced) SSL session parameters
[ https://issues.apache.org/jira/browse/FLINK-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-9879: -- Fix Version/s: (was: 1.12.0) 1.13.0 > Find sane defaults for (advanced) SSL session parameters > > > Key: FLINK-9879 > URL: https://issues.apache.org/jira/browse/FLINK-9879 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Affects Versions: 1.5.2, 1.6.0 >Reporter: Nico Kruber >Priority: Major > Fix For: 1.13.0 > > > After adding these configuration parameters with > https://issues.apache.org/jira/browse/FLINK-9878: > - SSL session cache size > - SSL session timeout > - SSL handshake timeout > - SSL close notify flush timeout > We should try to find sane defaults that "just work" :) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-11937) Resolve small file problem in RocksDB incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-11937: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Resolve small file problem in RocksDB incremental checkpoint > > > Key: FLINK-11937 > URL: https://issues.apache.org/jira/browse/FLINK-11937 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Congxian Qiu >Assignee: Congxian Qiu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently when incremental checkpoint is enabled in RocksDBStateBackend a > separate file will be generated on DFS for each sst file. This may cause > “file flood” when running intensive workload (many jobs with high > parallelism) in big cluster. According to our observation in Alibaba > production, such file flood introduces at lease two drawbacks when using HDFS > as the checkpoint storage FileSystem: 1) huge number of RPC request issued to > NN which may burst its response queue; 2) huge number of files causes big > pressure on NN’s on-heap memory. > In Flink we ever noticed similar small file flood problem and tried to > resolved it by introducing ByteStreamStateHandle(FLINK-2808), but this > solution has its limitation that if we configure the threshold too low there > will still be too many small files, while if too high the JM will finally > OOM, thus could hardly resolve the issue in case of using RocksDBStateBackend > with incremental snapshot strategy. > We propose a new OutputStream called > FileSegmentCheckpointStateOutputStream(FSCSOS) to fix the problem. FSCSOS > will reuse the same underlying distributed file until its size exceeds a > preset threshold. We > plan to complete the work in 3 steps: firstly introduce FSCSOS, secondly > resolve the specific storage amplification issue on FSCSOS, and lastly add an > option to reuse FSCSOS across multiple checkpoints to further reduce the DFS > file number. > More details please refer to the attached design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12273) The default value of CheckpointRetentionPolicy should be RETAIN_ON_FAILURE, otherwise it cannot be recovered according to checkpoint after failure.
[ https://issues.apache.org/jira/browse/FLINK-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-12273: --- Fix Version/s: (was: 1.12.0) 1.13.0 > The default value of CheckpointRetentionPolicy should be RETAIN_ON_FAILURE, > otherwise it cannot be recovered according to checkpoint after failure. > --- > > Key: FLINK-12273 > URL: https://issues.apache.org/jira/browse/FLINK-12273 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0 >Reporter: Mr.Nineteen >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The default value of CheckpointRetentionPolicy should be RETAIN_ON_FAILURE, > otherwise it cannot be recovered according to checkpoint after failure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12302) Fixed the wrong finalStatus of yarn application when application finished
[ https://issues.apache.org/jira/browse/FLINK-12302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-12302: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Fixed the wrong finalStatus of yarn application when application finished > - > > Key: FLINK-12302 > URL: https://issues.apache.org/jira/browse/FLINK-12302 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.8.0 >Reporter: lamber-ken >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: fix-bad-finalStatus.patch, flink-conf.yaml, > image-2019-04-23-19-56-49-933.png, image-2019-05-28-00-46-49-740.png, > image-2019-05-28-00-50-13-500.png, jobmanager-05-27.log, jobmanager-1.log, > jobmanager-2.log, screenshot-1.png, screenshot-2.png, > spslave4.bigdata.ly_23951, spslave5.bigdata.ly_20271, test.jar > > Time Spent: 20m > Remaining Estimate: 0h > > flink job(flink-1.6.3) failed in per-job yarn cluste mode, the > resourcemanager of yarn rerun the job. > when the job failed again, the application while finish, but the finalStatus > is +UNDEFINED,+ It's better to show state +FAILED+ > !image-2019-04-23-19-56-49-933.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12412) Allow ListTypeInfo used for java.util.List and MapTypeInfo used for java.util.Map
[
https://issues.apache.org/jira/browse/FLINK-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-12412:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Allow ListTypeInfo used for java.util.List and MapTypeInfo used for
> java.util.Map
> -
>
> Key: FLINK-12412
> URL: https://issues.apache.org/jira/browse/FLINK-12412
> Project: Flink
> Issue Type: New Feature
> Components: API / Type Serialization System
>Affects Versions: 1.9.0
>Reporter: YangFei
>Assignee: YangFei
>Priority: Major
> Labels: pull-request-available, starer
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
>
> {code:java}
> // code placeholder
> public static class UserBehavior {
> public long userId;
> public long itemId;
> public int categoryId;
> public long timestamp;
> public List comments;
> }
> public static void main(String[] args) throws Exception {
> PojoTypeInfo pojoType = (PojoTypeInfo)
> TypeExtractor.createTypeInfo(UserBehavior.class);
> }
> {code}
>
> The filed comments in UserBehavior will be extracted by TypeExtractor to
> GenericType .
> I think it can be extracted to ListTypeInfo .
> This would be a big improvement as in many cases classes including List or Map
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12520) Support to provide fully-qualified domain host name in TaskManagerMetricGroup
[
https://issues.apache.org/jira/browse/FLINK-12520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-12520:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Support to provide fully-qualified domain host name in TaskManagerMetricGroup
> -
>
> Key: FLINK-12520
> URL: https://issues.apache.org/jira/browse/FLINK-12520
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Inspired from Chinese
> [user-mail|https://lists.apache.org/thread.html/e1774a42430815b689ea792103d002b1da734d6086682d34c044ef35@%3Cuser-zh.flink.apache.org%3E]
> which complains that host name in metrics name could only show the first
> part. However, their full host name is like "{{ambari.host12.yy}}" which
> means the first part "{{ambari}}" cannot identify anything.
> We could support to let user record their full host name in the
> TaskManagerMetricGroup so that to identify metrics from different hosts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12590) Replace http links in documentation
[ https://issues.apache.org/jira/browse/FLINK-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-12590: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Replace http links in documentation > --- > > Key: FLINK-12590 > URL: https://issues.apache.org/jira/browse/FLINK-12590 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.7.2, 1.8.0, 1.9.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12692) Support disk spilling in HeapKeyedStateBackend
[
https://issues.apache.org/jira/browse/FLINK-12692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-12692:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Support disk spilling in HeapKeyedStateBackend
> --
>
> Key: FLINK-12692
> URL: https://issues.apache.org/jira/browse/FLINK-12692
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{HeapKeyedStateBackend}} is one of the two {{KeyedStateBackends}} in Flink,
> since state lives as Java objects on the heap and the de/serialization only
> happens during state snapshot and restore, it outperforms
> {{RocksDBKeyedStateBackend}} when all data could reside in memory.
> However, along with the advantage, {{HeapKeyedStateBackend}} also has its
> shortcomings, and the most painful one is the difficulty to estimate the
> maximum heap size (Xmx) to set, and we will suffer from GC impact once the
> heap memory is not enough to hold all state data. There’re several
> (inevitable) causes for such scenario, including (but not limited to):
> * Memory overhead of Java object representation (tens of times of the
> serialized data size).
> * Data flood caused by burst traffic.
> * Data accumulation caused by source malfunction.
> To resolve this problem, we propose a solution to support spilling state data
> to disk before heap memory is exhausted. We will monitor the heap usage and
> choose the coldest data to spill, and reload them when heap memory is
> regained after data removing or TTL expiration, automatically.
> More details please refer to the design doc and mailing list discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-12884) FLIP-144: Native Kubernetes HA Service
[ https://issues.apache.org/jira/browse/FLINK-12884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-12884: --- Fix Version/s: (was: 1.12.0) 1.13.0 > FLIP-144: Native Kubernetes HA Service > -- > > Key: FLINK-12884 > URL: https://issues.apache.org/jira/browse/FLINK-12884 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes, Runtime / Coordination >Reporter: MalcolmSanders >Assignee: Yang Wang >Priority: Major > Fix For: 1.13.0 > > > Currently flink only supports HighAvailabilityService using zookeeper. As a > result, it requires a zookeeper cluster to be deployed on k8s cluster if our > customers needs high availability for flink. If we support > HighAvailabilityService based on native k8s APIs, it will save the efforts of > zookeeper deployment as well as the resources used by zookeeper cluster. It > might be especially helpful for customers who run small-scale k8s clusters so > that flink HighAvailabilityService may not cause too much overhead on k8s > clusters. > Previously [FLINK-11105|https://issues.apache.org/jira/browse/FLINK-11105] > has proposed a HighAvailabilityService using etcd. As [~NathanHowell] > suggested in FLINK-11105, since k8s doesn't expose its own etcd cluster by > design (see [Securing etcd > clusters|https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#securing-etcd-clusters]), > it also requires the deployment of etcd cluster if flink uses etcd to > achieve HA. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13809) The log directory of Flink Python API is unwritable if it is installed via "sudo"
[
https://issues.apache.org/jira/browse/FLINK-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13809:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> The log directory of Flink Python API is unwritable if it is installed via
> "sudo"
> --
>
> Key: FLINK-13809
> URL: https://issues.apache.org/jira/browse/FLINK-13809
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.9.0
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, if the python apache-flink package is installed via "sudo", an
> exception will be thrown when starting the flink python shell:
> {code:java}
> log4j:ERROR setFile(null,false) call failed. java.io.FileNotFoundException:
> /Library/Python/2.7/site-packages/pyflink/log/flink-zhongwei-python-zhongweideMacBook-Pro.local.log
> (Permission denied) at java.io.FileOutputStream.open0(Native Method) at
> java.io.FileOutputStream.open(FileOutputStream.java:270) at
> java.io.FileOutputStream.(FileOutputStream.java:213) at
> java.io.FileOutputStream.(FileOutputStream.java:133) at
> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127) at
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at
> org.apache.flink.api.java.ExecutionEnvironment.(ExecutionEnvironment.java:102)
> at java.lang.Class.forName0(Native Method) at
> java.lang.Class.forName(Class.java:348) at
> org.apache.flink.api.python.shaded.py4j.reflection.CurrentThreadClassLoadingStrategy.classForName(CurrentThreadClassLoadingStrategy.java:40)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionUtil.classForName(ReflectionUtil.java:51)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.forName(TypeUtil.java:243)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:175)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:87)
> at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It does not affect the running of flink python shell but it would be better
> if we can fix it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13149) improve flink ui to make it easier to be used by other projects
[
https://issues.apache.org/jira/browse/FLINK-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13149:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> improve flink ui to make it easier to be used by other projects
> ---
>
> Key: FLINK-13149
> URL: https://issues.apache.org/jira/browse/FLINK-13149
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0
>Reporter: Jie TANG
>Priority: Major
> Labels: performance, pull-request-available
> Fix For: 1.13.0
>
> Attachments: 20190709113558.jpg, snapshot-1.png, snapshot-2.png
>
> Original Estimate: 120h
> Time Spent: 10m
> Remaining Estimate: 119h 50m
>
> The new web UI looks nice, but there are still some problems when I try to
> integrate it into the other frontend project, I think we can make some
> changes in order to make it easier to be customized.
> !20190709113558.jpg|width=596,height=185!
> *These changes will not bring break changes and it will also not affect the
> user interface.*
> * Migrate the code for {{app.config.ts}} to {{config.service.ts}} to make it
> configurable in angular DI system.
> * Add property named {{JOB_PREFIX}} to {{config.service.ts}} for
> {{job.service.ts}} and {{metrics.service.ts}} to make them configurable
> * Add optional param to the url to hide menu and header to make it possible
> for users want to embed the flink ui as an iframe in other website
> * Update Angular version to 8.0 (no break changes)
> * set `paths` of `tsconfig.json` ("@flink-runtime-web/*": ["./src/app/*",
> "./src/environments/*"]) to ensure that it does not have the same name as the
> personal project path.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13976) Make modern kafka E2E test runnable on Java 11
[ https://issues.apache.org/jira/browse/FLINK-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-13976: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Make modern kafka E2E test runnable on Java 11 > -- > > Key: FLINK-13976 > URL: https://issues.apache.org/jira/browse/FLINK-13976 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.13.0 > > > The modern kafka E2E tests currently fails on Java 11. We should find a way > to make this oe runnable at least. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags
[
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13418:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it
> would involve all variables as tags (please see code
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
> However, user could adjust their own scope format to abort unnecessary
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other
> words, InfluxDB need tag-key and tag-value to compose as its tags while we
> could only get all variables (without any filter acording to scope format) or
> only scopeComponents (could be treated as tag-value). I think that's why
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the
> overall series in InfluxDB, it would never be a good idea to contain too many
> tags, not to mention the [default value of series per
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
> is only one million.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13692) Make CompletedCheckpointStore backwards compatible?
[
https://issues.apache.org/jira/browse/FLINK-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13692:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Make CompletedCheckpointStore backwards compatible?
> ---
>
> Key: FLINK-13692
> URL: https://issues.apache.org/jira/browse/FLINK-13692
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.9.4, 1.13.0
>
>
> With FLINK-11159 we changed the signature of
> {{CompletedCheckpointStore#getLatestCheckpoint}} from
> {{getLatestCheckpoint()}} to {{getLatestCheckpoint(boolean)}}. This is an API
> breaking change which causes users of this interface having to adapt their
> implementation and recompile it. Since {{CompletedCheckpointStore}} is no
> public API, this change is technically ok, but not so nice.
> If we decide to provide backwards compatibility, then we should still keep
> {{getLatestCheckpoint()}} and add a deprecation annotation to it. At least,
> we should update the JavaDocs of the
> {{CompletedCheckpointStore#getLatestCheckpoint}} method to reflect what the
> passed parameter states.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13698) Rework threading model of CheckpointCoordinator
[
https://issues.apache.org/jira/browse/FLINK-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13698:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Rework threading model of CheckpointCoordinator
> ---
>
> Key: FLINK-13698
> URL: https://issues.apache.org/jira/browse/FLINK-13698
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Biao Liu
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently {{CheckpointCoordinator}} and {{CheckpointFailureManager}} code is
> executed by multiple different threads (mostly {{ioExecutor}}, but not only).
> It's causing multiple concurrency issues, for example:
> https://issues.apache.org/jira/browse/FLINK-13497
> Proper fix would be to rethink threading model there. At first glance it
> doesn't seem that this code should be multi threaded, except of parts doing
> the actual IO operations, so it should be possible to run everything in one
> single ExecutionGraph's thread and just run asynchronously necessary IO
> operations with some feedback loop ("mailbox style").
> I would strongly recommend fixing this issue before adding new features in
> the \{{CheckpointCoordinator}} component.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14106) Make SlotManager pluggable
[ https://issues.apache.org/jira/browse/FLINK-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14106: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Make SlotManager pluggable > -- > > Key: FLINK-14106 > URL: https://issues.apache.org/jira/browse/FLINK-14106 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Fix For: 1.13.0 > > > As we are enabling fine grained resource management in 1.10, we can have > various resource scheduling strategies. Such strategies generally should make > the following three decisions. > * When to launch new / release existing TMs? (How many TMs) > * What and how many resources should TMs be started with? > * How to allocate between slot requests and TM resources? > We may want to make above decisions differently in different scenarios > (active/reactive mode, perjob/session mode, etc.). Therefore, we propose to > make the scheduling strategies pluggable. > We propose to make the following changes: > * Make SlotManager an interface, and implements it differently for different > strategies strategies. > * Modify ResourceManager-SlotManager interfaces to cover all the three > decisions mentioned above in SlotManager. In particular, SlotManager needs to > allocate TM resources instead of slot resources from ResourceActions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13795) Web UI logs errors when selecting Checkpoint Tab for Batch Jobs
[
https://issues.apache.org/jira/browse/FLINK-13795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13795:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Web UI logs errors when selecting Checkpoint Tab for Batch Jobs
> ---
>
> Key: FLINK-13795
> URL: https://issues.apache.org/jira/browse/FLINK-13795
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 1.13.0
>
>
> The logs of the REST endpoint print errors if you run a batch job and then
> select the "Checkpoints" tab.
> I would expect that this simply shows "no checkpoints available for this job"
> and not that an {{ERROR}} level entry appears in the log.
> {code}
> 2019-08-20 12:04:54,195 ERROR
> org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler
> - Exception occurred in REST handler: Checkpointing has not been enabled.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
[
https://issues.apache.org/jira/browse/FLINK-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13910:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Many serializable classes have no explicit 'serialVersionUID'
> -
>
> Key: FLINK-13910
> URL: https://issues.apache.org/jira/browse/FLINK-13910
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Attachments: SerializableNoSerialVersionUIDField,
> classes-without-uid-per-module, serializable-classes-without-uid-5249249
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, many serializable classes in Flink have no explicit
> 'serialVersionUID'. As [official
> doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
> said, {{Serializable classes must define a Serial Version UID}}.
> No 'serialVersionUID' would cause compatibility problem. Take
> {{TwoPhaseCommitSinkFunction}} for example, since no explicit
> 'serialVersionUID' defined, after
> [FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
> introduced, its default 'serialVersionUID' has changed from
> "4584405056408828651" to "4064406918549730832". In other words, if we submit
> a job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the
> usage of {{TwoPhaseCommitSinkFunction}}, we would get exception like:
> {code:java}
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> instantiate user function.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;
> local class incompatible: stream classdesc serialVersionUID =
> 4584405056408828651, local class serialVersionUID = 4064406918549730832
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:537)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:512)
> at
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:473)
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
> ... 4 more
> {code}
> Similar problems existed in
> {{org.apache.flink.streaming.api.operators.SimpleOperatorFactory}} which has
> different 'serialVersionUID' from release-1.9 and current master branch.
> IMO, we might have two options to fix this bug:
> # Add explicit serialVersionUID for those classes which is identical to
> latest Flink-1.9.0 release code.
> # Use similar mechanism like {{FailureTolerantObjectInputStream}} in
> {{InstantiationUtil}} to ignore serialVersionUID mismatch.
> I have collected all production classes without serialVersionUID from latest
> master branch in the attachment, which counts to 639 classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13719) Update Yarn E2E test docker image to run on Java 11
[ https://issues.apache.org/jira/browse/FLINK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-13719: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Update Yarn E2E test docker image to run on Java 11 > --- > > Key: FLINK-13719 > URL: https://issues.apache.org/jira/browse/FLINK-13719 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN, Tests >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.13.0 > > > The docker image used in the e2e tests is using Java 8. We have to setup > another build that uses Java 11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14191) Extend SlotManager to support dynamic slot allocation on pending task executors
[ https://issues.apache.org/jira/browse/FLINK-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14191: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Extend SlotManager to support dynamic slot allocation on pending task > executors > --- > > Key: FLINK-14191 > URL: https://issues.apache.org/jira/browse/FLINK-14191 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Fix For: 1.13.0 > > > * Introduce PendingTaskManagerResources > * Create PendingTaskManagerSlot on allocation, from > PendingTaskManagerResource > * Map registered task executors to matching PendingTaskManagerResources, and > allocate slots for corresponding PendingTaskManagerSlots > Convert registered task executor free slots into equivalent available > resources according to default slot resource profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13876) Remove ExecutionConfig field from PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-13876:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Remove ExecutionConfig field from PojoSerializer
>
>
> Key: FLINK-13876
> URL: https://issues.apache.org/jira/browse/FLINK-13876
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Fix For: 1.13.0
>
>
> The PojoSerializers stores an instance of ExecutionConfig as internal field,
> even though that the only information it may ever need are the registered
> kryo serializers.
> This has a few drawbacks:
> * It blocks the evolution of {{ExecutionConfig}} as serializers where stored
> in a state. Therefore any change to ExecutionConfig must be backwards
> compatible in respect to java serialization
> * It probably already introduced a bug, as upon restore the Snapshot actually
> recreates the serializer with an empty ExecutionConfig (see
> org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshot#restoreSerializer)
> I suggest to remove the field completely and adjust corresponding usages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14187) FLIP-56 Dynamic Slot Allocation
[ https://issues.apache.org/jira/browse/FLINK-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14187: --- Fix Version/s: (was: 1.12.0) 1.13.0 > FLIP-56 Dynamic Slot Allocation > --- > > Key: FLINK-14187 > URL: https://issues.apache.org/jira/browse/FLINK-14187 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Affects Versions: 1.9.0 >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Labels: Umbrella > Fix For: 1.13.0 > > > This is the umbrella issue for 'FLIP-56: Dynamic Slot Allocation'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14194) Clean-up of legacy mode.
[ https://issues.apache.org/jira/browse/FLINK-14194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14194: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Clean-up of legacy mode. > > > Key: FLINK-14194 > URL: https://issues.apache.org/jira/browse/FLINK-14194 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14032) Make the cache size of RocksDBPriorityQueueSetFactory configurable
[
https://issues.apache.org/jira/browse/FLINK-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14032:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Make the cache size of RocksDBPriorityQueueSetFactory configurable
> --
>
> Key: FLINK-14032
> URL: https://issues.apache.org/jira/browse/FLINK-14032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
> Labels: usability
> Fix For: 1.13.0
>
>
> Currently, the cache size of {{RocksDBPriorityQueueSetFactory}} has been set
> as 128 and no any ways to configure this to other value. (We could increase
> this to obtain better performance if necessary). Actually, this is also a
> TODO for quiet a long time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14023) Support accessing job parameters in Python user-defined functions
[ https://issues.apache.org/jira/browse/FLINK-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14023: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Support accessing job parameters in Python user-defined functions > - > > Key: FLINK-14023 > URL: https://issues.apache.org/jira/browse/FLINK-14023 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Dian Fu >Priority: Major > Fix For: 1.13.0 > > > Currently, it’s possible to access job parameters in the Java user-defined > functions. It could be used to define the behavior according to job > parameters. It should also be supported for Python user-defined functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14127) Better BackPressure Detection in WebUI
[ https://issues.apache.org/jira/browse/FLINK-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14127: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Better BackPressure Detection in WebUI > -- > > Key: FLINK-14127 > URL: https://issues.apache.org/jira/browse/FLINK-14127 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Yadong Xie >Assignee: Yadong Xie >Priority: Major > Fix For: 1.13.0 > > Attachments: 屏幕快照 2019-09-19 下午6.00.05.png, 屏幕快照 2019-09-19 > 下午6.00.57.png, 屏幕快照 2019-09-19 下午6.01.43.png > > > According to the > [Document|https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/back_pressure.html], > the backpressure monitor only triggered on request and it is currently not > available via metrics. This means that in the web UI we have no way to show > all the backpressure state of all vertexes at the same time. The users need > to click every vertex to get its backpressure state. > !屏幕快照 2019-09-19 下午6.00.05.png|width=510,height=197! > In Flink 1.9.0 and above, there are four metrics available(outPoolUsage, > inPoolUsage, floatingBuffersUsage, exclusiveBuffersUsage), we can use these > metrics to determine if there are possible backpressure, and then use the > backpressure REST API to confirm it. > Here is a table get from > [https://flink.apache.org/2019/07/23/flink-network-stack-2.html] > !屏幕快照 2019-09-19 下午6.00.57.png|width=516,height=304! > > We can display the possible backpressure status on the vertex graph, thus > users can get all the vertex backpressure states and locate the potential > problem quickly. > > !屏幕快照 2019-09-19 下午6.01.43.png|width=572,height=277! > > REST API needed: > add outPoolUsage, inPoolUsage, floatingBuffersUsage, exclusiveBuffersUsage > metrics for each vertex in the /jobs/:jobId API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14122) Extend State Processor API to read ListCheckpointed operator state
[ https://issues.apache.org/jira/browse/FLINK-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14122: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Extend State Processor API to read ListCheckpointed operator state > -- > > Key: FLINK-14122 > URL: https://issues.apache.org/jira/browse/FLINK-14122 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Seth Wiesman >Priority: Major > Labels: usability > Fix For: 1.13.0 > > > The state processor api cannot read operator state using the > ListCheckpointed interface because it requires access the JavaSerializer > which is package private. Instead of making that class public, we should > offer a readListCheckpointed Method to easily read this state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14125) Display memory and CPU usage in the overview page
[ https://issues.apache.org/jira/browse/FLINK-14125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14125: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Display memory and CPU usage in the overview page > - > > Key: FLINK-14125 > URL: https://issues.apache.org/jira/browse/FLINK-14125 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Yadong Xie >Priority: Major > Fix For: 1.13.0 > > Attachments: 屏幕快照 2019-09-19 下午5.03.14.png > > > In the overview page of Web UI, besides the task slots and jobs, we could add > memory and CPU usage metrics to the cluster-level, these metrics are already > available in the Blink branch. > !屏幕快照 2019-09-19 下午5.03.14.png|width=564,height=283! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14820) Replace Time with Duration in FutureUtils
[ https://issues.apache.org/jira/browse/FLINK-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14820: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Replace Time with Duration in FutureUtils > - > > Key: FLINK-14820 > URL: https://issues.apache.org/jira/browse/FLINK-14820 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14028) Support to configure the log level of the Python user-defined functions
[ https://issues.apache.org/jira/browse/FLINK-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14028: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Support to configure the log level of the Python user-defined functions > --- > > Key: FLINK-14028 > URL: https://issues.apache.org/jira/browse/FLINK-14028 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Dian Fu >Priority: Major > Fix For: 1.13.0 > > > Beam's portability framework has provided the ability to collecting log to > operator from the Python workers. Currently the log level INFO is set by > default. It should support to configure the log level of the Python > user-defined functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14340) Specify an unique DFSClient name for Hadoop FileSystem
[
https://issues.apache.org/jira/browse/FLINK-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14340:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Specify an unique DFSClient name for Hadoop FileSystem
> --
>
> Key: FLINK-14340
> URL: https://issues.apache.org/jira/browse/FLINK-14340
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
>Reporter: Congxian Qiu
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, when Flink read/write to HDFS, we do not set the DFSClient name
> for all the connections, so we can’t distinguish the connections, and can’t
> find the specific Job or TM quickly.
> This issue wants to add the {{container_id}} as a unique name when init
> Hadoop File System, so we can easily distinguish the connections belongs to
> which Job/TM.
>
> Core changes is add a line such as below in
> {{org.apache.flink.runtime.fs.hdfs.HadoopFsFactory#create}}
>
> {code:java}
> hadoopConfig.set(“mapreduce.task.attempt.id”,
> System.getenv().getOrDefault(CONTAINER_KEY_IN_ENV,
> DEFAULT_CONTAINER_ID));{code}
>
> Currently, In {{YarnResourceManager}} and {{MesosResourceManager}} we both
> have an enviroment key {{ENV_FLINK_CONTAINER_ID = "_FLINK_CONTAINER_ID"}}, so
> maybe we should introduce this key in {{StandaloneResourceManager}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14684: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Assignee: Jiawei Wu >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14159:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.3, 1.13.0, 1.11.4
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14971) Make all the non-IO operations in CheckpointCoordinator single-threaded
[
https://issues.apache.org/jira/browse/FLINK-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14971:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Make all the non-IO operations in CheckpointCoordinator single-threaded
> ---
>
> Key: FLINK-14971
> URL: https://issues.apache.org/jira/browse/FLINK-14971
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the ACK and declined message handling are executed in IO thread.
> This is the only rest part that non-IO operations are executed in IO thread.
> It blocks introducing main thread executor for {{CheckpointCoordinator}}. It
> would be resolved in this task.
> After resolving the ACK and declined message issue, the main thread executor
> would be introduced into {{CheckpointCoordinator}} to instead of timer
> thread. However the timer thread would be kept (maybe for a while
> temporarily) to schedule periodic triggering, since FLINK-13848 is not
> accepted yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14190) Extend SlotManager to support dynamic slot allocation on registered TaskExecutors.
[ https://issues.apache.org/jira/browse/FLINK-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14190: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Extend SlotManager to support dynamic slot allocation on registered > TaskExecutors. > -- > > Key: FLINK-14190 > URL: https://issues.apache.org/jira/browse/FLINK-14190 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Fix For: 1.13.0 > > > * Bookkeep task manager available resources > * Match between slot requests and task executor resources > ** Find task executors with matching available resources for slot requests > ** Find matching pending slot requests for task executors with new available > resources > * Create TaskManagerSlot on allocation and remove on free. > * Request slot from TaskExecutor with resource profiles. > Use RM calculated default resource profiles for all slot requests. Convert > free slots in SlotReports into equivalent available resources according to > default slot resource profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14920) Set up environment to run performance e2e tests
[ https://issues.apache.org/jira/browse/FLINK-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14920: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Set up environment to run performance e2e tests > --- > > Key: FLINK-14920 > URL: https://issues.apache.org/jira/browse/FLINK-14920 > Project: Flink > Issue Type: Sub-task > Components: Test Infrastructure >Reporter: Yu Li >Assignee: Aihua Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As proposed in > [FLIP-83|https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework], > we need to complete below tasks here: > * Prepare a small cluster and setup environments to run the tests > * Setup Jenkins to trigger the performance e2e tests > * Report the result to [code-speed center|http://codespeed.dak8s.net:8000] to > show the comparison. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14192) Enable the dynamic slot allocation feature.
[ https://issues.apache.org/jira/browse/FLINK-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14192: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Enable the dynamic slot allocation feature. > --- > > Key: FLINK-14192 > URL: https://issues.apache.org/jira/browse/FLINK-14192 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Priority: Major > Fix For: 1.13.0 > > > * ResourceManager uses TaskExecutor registered default slot resource > profiles, instead of that calculated on RM side. > * ResourceManager uses actual requested resource profiles for slot requests, > instead assuming default profile for all requests. > * TaskExecutor bookkeep with requested resource profiles instead, instead of > assuming default profile for all requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14193) Update RestAPI / Web UI
[ https://issues.apache.org/jira/browse/FLINK-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14193: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Update RestAPI / Web UI > --- > > Key: FLINK-14193 > URL: https://issues.apache.org/jira/browse/FLINK-14193 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Web Frontend >Reporter: Xintong Song >Priority: Major > Fix For: 1.13.0 > > > * Update RestAPI / WebUI to properly display information of available > resources and allocated slots of task executors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14222) Optimize for Python UDFs with all parameters are constant values
[ https://issues.apache.org/jira/browse/FLINK-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14222: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Optimize for Python UDFs with all parameters are constant values > > > Key: FLINK-14222 > URL: https://issues.apache.org/jira/browse/FLINK-14222 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Dian Fu >Priority: Major > Fix For: 1.13.0 > > > As discussed in [https://github.com/apache/flink/pull/9766], The Python UDFs > could be optimized to a constant value if it is deterministic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14867) Move TextInputFormat & TextOutputFormat to flink-core
[
https://issues.apache.org/jira/browse/FLINK-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14867:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Move TextInputFormat & TextOutputFormat to flink-core
> -
>
> Key: FLINK-14867
> URL: https://issues.apache.org/jira/browse/FLINK-14867
> Project: Flink
> Issue Type: Improvement
> Components: API / DataSet, API / DataStream
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is one step to decouple the dependency from flink-streaming-java to
> flink-java. We already have a package {{o.a.f.core.io}} for these formats.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14479) Strange exceptions found in log file after executing `test_udf.py`
[
https://issues.apache.org/jira/browse/FLINK-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14479:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Strange exceptions found in log file after executing `test_udf.py`
> --
>
> Key: FLINK-14479
> URL: https://issues.apache.org/jira/browse/FLINK-14479
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Reporter: sunjincheng
>Assignee: Wei Zhong
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: screenshot-1.png
>
>
> There are several strange exceptions as follow in
> `${flink_source}/build-target/log/flink-${username}-python-udf-boot-${machine_name}.local.log`
> after executing
> `${flink_source}/flink-python/pyflink/table/tests/test_udf.py`:
> Traceback (most recent call last):
> {code:java}
> File
> "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py",
> line 193, in _run_module_as_main
> "__main__", mod_spec)
> File
> "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py",
> line 85, in _run_code
> exec(code, run_globals)
> File
> "/Users/zhongwei/flink/flink-python/pyflink/fn_execution/sdk_worker_main.py",
> line 30, in
> apache_beam.runners.worker.sdk_worker_main.main(sys.argv)
> File
> "/Users/zhongwei/pyflink_env/py37/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 148, in main
> sdk_pipeline_options.view_as(pipeline_options.ProfilingOptions))
> File
> "/Users/zhongwei/pyflink_env/py37/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 133, in run
> for work_request in control_stub.Control(get_responses()):
> File
> "/Users/zhongwei/pyflink_env/py37/lib/python3.7/site-packages/grpc/_channel.py",
> line 364, in __next__
> return self._next()
> File
> "/Users/zhongwei/pyflink_env/py37/lib/python3.7/site-packages/grpc/_channel.py",
> line 347, in _next
> raise self
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.CANCELLED
> details = "Runner closed connection"
> debug_error_string =
> "{"created":"@1571660342.057172000","description":"Error received from peer
> ipv6:[::1]:52699","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Runner
> closed connection","grpc_status":1}"{code}
> It appears randomly when executing test cases of blink planner. Although it
> does not affect test results we need to find out why it appears.
> Welcome any feedback!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14350) Introduce dedicated MetricScope
[ https://issues.apache.org/jira/browse/FLINK-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14350: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Introduce dedicated MetricScope > --- > > Key: FLINK-14350 > URL: https://issues.apache.org/jira/browse/FLINK-14350 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The MetricGroup interface is currently overloaded, being used both for > registering groups/metrics (typically called from user-functions) and > accessing scope information (typically called from reporters). > Due to effectively having 2 different audiences with different use-cases it > makes sense to move the scope-related methods into a separate interface. > This should make it a lot easier to extend these interfaces, as significantly > less tests have to be adjusted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14351) Refactor MetricRegistry delimiter retrieval into separate interface
[ https://issues.apache.org/jira/browse/FLINK-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14351: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Refactor MetricRegistry delimiter retrieval into separate interface > --- > > Key: FLINK-14351 > URL: https://issues.apache.org/jira/browse/FLINK-14351 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The MetricRegistry offers a few methods for retrieving configured delimiters, > which are used a fair bit during scope operations; however other methods > aren't being used in these contexts. > Hence we could reduce access and simplify testing by introducing a dedicated > interface for these methods that the registry extends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14707) Refactor checkpoint related methods within Environment
[
https://issues.apache.org/jira/browse/FLINK-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14707:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Refactor checkpoint related methods within Environment
> --
>
> Key: FLINK-14707
> URL: https://issues.apache.org/jira/browse/FLINK-14707
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.13.0
>
>
> After FLINK-7720 fixed, the two methods of
> {{Environment#acknowledgeCheckpoint}} have been actually useless, and their
> usage have been overridden by {{TaskStateManager#reportTaskStateSnapshots}}.
> More generally, we would consider {{Environment}} interface as getter to
> access components instead of doing real actions. I prefer to remove
> {{acknowledgeCheckpoint}} and {{declineCheckpoint}} methods or at least
> deprecated them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14919) Add performance e2e test suite for basic operations
[ https://issues.apache.org/jira/browse/FLINK-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14919: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Add performance e2e test suite for basic operations > --- > > Key: FLINK-14919 > URL: https://issues.apache.org/jira/browse/FLINK-14919 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Yu Li >Assignee: Aihua Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add the test suite for basic Flink operations as proposed in > [FLIP-83|https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework]. > Target at completing the work before 1.10 so we could use it to guard the > release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14379) Supplement documentation about raw state
[ https://issues.apache.org/jira/browse/FLINK-14379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14379: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Supplement documentation about raw state > > > Key: FLINK-14379 > URL: https://issues.apache.org/jira/browse/FLINK-14379 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Fix For: 1.13.0 > > > Currently, we only have very simple > [documentation|https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state] > or even we could say only one sentence to talk about raw state. It might > lead beginner of Flink feel not so clear about this concept, I think we > should supplement documentation about raw state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14383) Support python UDFs with constant value of time interval types
[ https://issues.apache.org/jira/browse/FLINK-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14383: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Support python UDFs with constant value of time interval types > -- > > Key: FLINK-14383 > URL: https://issues.apache.org/jira/browse/FLINK-14383 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Huang Xingbo >Priority: Major > Fix For: 1.13.0 > > > As discussed > [here|https://github.com/apache/flink/pull/9858#issuecomment-541312088], this > issue is dedicated to add support for python UDFs with constant value of time > interval types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14393) add an option to enable/disable cancel job in web ui
[ https://issues.apache.org/jira/browse/FLINK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14393: --- Fix Version/s: (was: 1.12.0) 1.13.0 > add an option to enable/disable cancel job in web ui > > > Key: FLINK-14393 > URL: https://issues.apache.org/jira/browse/FLINK-14393 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Yadong Xie >Priority: Major > Fix For: 1.13.0 > > > add the option to enable/disable cancel job in web ui > when disabled, user can not cancel a job through the web ui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14471) Hide error message when metric api failed
[ https://issues.apache.org/jira/browse/FLINK-14471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14471: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Hide error message when metric api failed > - > > Key: FLINK-14471 > URL: https://issues.apache.org/jira/browse/FLINK-14471 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.9.1 >Reporter: Yadong Xie >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The error message should hide when metric api failed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14410) Retrieve CharacterFilter once after registration
[
https://issues.apache.org/jira/browse/FLINK-14410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14410:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Retrieve CharacterFilter once after registration
>
>
> Key: FLINK-14410
> URL: https://issues.apache.org/jira/browse/FLINK-14410
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.13.0
>
>
> Reporters can use a {{CharacterFilter}} to remove/replace characters when
> assembling metric identifiers. Currently the reporter pass this filter in
> every call to scope-related methods on the {{MetricGrou}}. We could
> streamline the {{MetricScope}} API proposed in FLINK-14350 if instead we
> retrieved this filter once after the reporter is initialized.
> This would furthermore prevent subtle bugs if reporters use varying filters;
> due to the caching of scopes only the first filter would truly be applied to
> the scope.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14500) Support Flink Python User-Defined Stateless Function for Table - Phase 2
[ https://issues.apache.org/jira/browse/FLINK-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14500: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Support Flink Python User-Defined Stateless Function for Table - Phase 2 > > > Key: FLINK-14500 > URL: https://issues.apache.org/jira/browse/FLINK-14500 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Dian Fu >Assignee: Huang Xingbo >Priority: Major > Fix For: 1.13.0 > > > This is the umbrella Jira which tracks the functionalities of "Python > User-Defined Stateless Function for Table" which are planned to be supported > in 1.11, such as docker mode support, user-defined metrics support, arrow > support, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15325) Input location preference which affects task distribution may make certain job performance worse
[ https://issues.apache.org/jira/browse/FLINK-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-15325: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Input location preference which affects task distribution may make certain > job performance worse > - > > Key: FLINK-15325 > URL: https://issues.apache.org/jira/browse/FLINK-15325 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Priority: Major > Fix For: 1.13.0 > > Attachments: D58ADB03-7187-46B1-B077-91E5005FD463.png > > > When running TPC-DS jobs in a session cluster, we observed that sometimes > tasks are not evenly distributed in TMs. The root cause turned out to be that > the downstream tasks tend to be TM or host local with its input tasks. This > helps to reduce network shuffle. > However, in certain cases, like the topology presented in the attached image, > jamming the input task's TM and machine with downstream tasks would affect > the performance. In this case, respecting input location preferences is > causing troubles more than bringing benefits. > So I'm wondering whether we should introduce a config so that users can > disable input location preferences? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15018) Add event time page to Concepts
[ https://issues.apache.org/jira/browse/FLINK-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-15018: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Add event time page to Concepts > --- > > Key: FLINK-15018 > URL: https://issues.apache.org/jira/browse/FLINK-15018 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.10.3, 1.11.3, 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We should disentangle the conceptual overview of event time from the > DataStream documentation so that it can be used by DataStream and SQL users. > This has the added benefit of making the page more stable as it will not have > to be rewritten or touched every time an API is changed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15000) WebUI Metrics is very slow in large parallelism
[
https://issues.apache.org/jira/browse/FLINK-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15000:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> WebUI Metrics is very slow in large parallelism
> ---
>
> Key: FLINK-15000
> URL: https://issues.apache.org/jira/browse/FLINK-15000
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0, 1.9.1
>Reporter: fa zheng
>Assignee: fa zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> metrics in web ui are very slow when parallelism is huge. It's hard to add
> metric and choose one metric. I run carTopSpeedWindowingExample with command
> {code:java}
> //代码占位符
> flink run -m yarn-cluster -p 1200 examples/streaming/TopSpeedWindowing.jar
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15262) kafka connector doesn't read from beginning immediately when 'connector.startup-mode' = 'earliest-offset'
[
https://issues.apache.org/jira/browse/FLINK-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15262:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> kafka connector doesn't read from beginning immediately when
> 'connector.startup-mode' = 'earliest-offset'
> --
>
> Key: FLINK-15262
> URL: https://issues.apache.org/jira/browse/FLINK-15262
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0
>Reporter: Bowen Li
>Assignee: Jiangjie Qin
>Priority: Major
> Fix For: 1.10.3, 1.13.0, 1.11.4
>
>
> I created a kafka table in Flink to read from my kakfa topic (already has
> messages in it) in earliest offset, but `select * from test` query in Flink
> doesn't start to read until a new message comes. If no new message arrives,
> the query just sit there and never produce result.
> What I expect is that the query should immediate produce result on all
> existing message without having to wait for a new message to "trigger" data
> processing.
> DDL that I used according to DDL document at
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#kafka-connector
> {code:java}
> create table test(name String) with (
>'connector.type' = 'kafka',
>'connector.version' = 'universal',
>'connector.topic' = 'test',
>'connector.properties.zookeeper.connect' = 'localhost:2181',
>'connector.properties.bootstrap.servers' = 'localhost:9092',
>'connector.startup-mode' = 'earliest-offset',
>'format.type' = 'csv',
>'update-mode' = 'append'
> );
> {code}
> repro steps:
> 1) start a local kafka cluster following https://kafka.apache.org/quickstart
> with a topic named "test"
> 2) produce some records in kafka with simple strings as "john", "marry", etc,
> into the topic
> 3) start flink sql cli, add kafka cli dependency, create a Flink table as
> {code:java}
> create table test(name String) with (
>'connector.type' = 'kafka',
>'connector.version' = 'universal',
>'connector.topic' = 'test',
>'connector.properties.zookeeper.connect' = 'localhost:2181',
>'connector.properties.bootstrap.servers' = 'localhost:9092',
>'connector.startup-mode' = 'earliest-offset',
>'format.type' = 'csv'
> );
> {code}
> 4) run "select * from test" in SQL CLI
> Expected: upon running the query, we should immediately see records already
> in kafka, like "john" and "marry"
> Reality: upon running the query, no record shows up. we have to produce some
> new records like "kitty" into the kafka topic to be able to see old records
> "john" and "marry"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15471) HA e2e check for empty .out files does not print specific error
[
https://issues.apache.org/jira/browse/FLINK-15471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15471:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> HA e2e check for empty .out files does not print specific error
> ---
>
> Key: FLINK-15471
> URL: https://issues.apache.org/jira/browse/FLINK-15471
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.10.3, 1.13.0, 1.11.4
>
>
> {{common_ha.sh#verify_logs:}}
> {code}
> if ! check_logs_for_non_empty_out_files; then
> echo "FAILURE: Alerts found at the general purpose job."
> EXIT_CODE=1
> fi
> {code}
> Since check_logs_for_non_empty_out_files does only set EXIT_CODE without
> modifying the return value the check will never fail.
> While the test will still fail (since EXIT_CODE is later being evaluated), we
> may not actually print the error cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15122) Reusing record object in StreamTaskNetworkInput
[ https://issues.apache.org/jira/browse/FLINK-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-15122: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Reusing record object in StreamTaskNetworkInput > --- > > Key: FLINK-15122 > URL: https://issues.apache.org/jira/browse/FLINK-15122 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.13.0 > > > Now blink's batch is forced to open object reusing, but the data read from > the network is not reused, which will lead to a large GC of the batch job. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15351) develop PostgresCatalog to connect Flink with Postgres tables and ecosystem
[ https://issues.apache.org/jira/browse/FLINK-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-15351: --- Fix Version/s: (was: 1.12.0) 1.13.0 > develop PostgresCatalog to connect Flink with Postgres tables and ecosystem > --- > > Key: FLINK-15351 > URL: https://issues.apache.org/jira/browse/FLINK-15351 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14918) Add performance e2e test module and scripts
[ https://issues.apache.org/jira/browse/FLINK-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14918: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Add performance e2e test module and scripts > --- > > Key: FLINK-14918 > URL: https://issues.apache.org/jira/browse/FLINK-14918 > Project: Flink > Issue Type: Sub-task > Components: Test Infrastructure >Reporter: Yu Li >Assignee: Aihua Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > As proposed in FLIP-83, create a separate directory/module in parallel with > flink-end-to-end-tests, with the name of flink-end-to-end-perf-tests, and add > necessary scripts to form the framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15586) BucketingSink is ignoring plugins when trying to re-instantiate the HadoopFileSystem
[
https://issues.apache.org/jira/browse/FLINK-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15586:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> BucketingSink is ignoring plugins when trying to re-instantiate the
> HadoopFileSystem
>
>
> Key: FLINK-15586
> URL: https://issues.apache.org/jira/browse/FLINK-15586
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Priority: Minor
> Fix For: 1.13.0
>
>
> {{BucketingSink#createHadoopFileSystem}} first is loading
> {{org.apache.flink.core.fs.FileSystem}} through the plugin class loader
> (correctly), but later in the "re-instantiate" branch it's ignoring the
> plugin class loader/{{PluginManager}} classes altogether, which means it will
> be looking for non existing classes in the parent class loader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14920) Set up environment to run performance e2e tests
[ https://issues.apache.org/jira/browse/FLINK-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14920: --- Fix Version/s: (was: 1.12.0) > Set up environment to run performance e2e tests > --- > > Key: FLINK-14920 > URL: https://issues.apache.org/jira/browse/FLINK-14920 > Project: Flink > Issue Type: Sub-task > Components: Test Infrastructure >Reporter: Yu Li >Assignee: Aihua Li >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > As proposed in > [FLIP-83|https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework], > we need to complete below tasks here: > * Prepare a small cluster and setup environments to run the tests > * Setup Jenkins to trigger the performance e2e tests > * Report the result to [code-speed center|http://codespeed.dak8s.net:8000] to > show the comparison. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15130) Drop "RequiredParameters" and "Options"
[ https://issues.apache.org/jira/browse/FLINK-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-15130: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Drop "RequiredParameters" and "Options" > --- > > Key: FLINK-15130 > URL: https://issues.apache.org/jira/browse/FLINK-15130 > Project: Flink > Issue Type: Task > Components: API / DataSet, API / DataStream >Affects Versions: 1.9.1 >Reporter: Stephan Ewen >Priority: Minor > Fix For: 1.13.0 > > > As per mailing list discussion, we want to drop those because they are unused > redundant code. > There are many options for command line parsing, including one in Flink > (Parameter Tool). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14934) Remove error log statement in ES connector
[ https://issues.apache.org/jira/browse/FLINK-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-14934: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Remove error log statement in ES connector > -- > > Key: FLINK-14934 > URL: https://issues.apache.org/jira/browse/FLINK-14934 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.9.1 >Reporter: Arvid Heise >Priority: Major > Fix For: 1.13.0 > > > The ES connector currently uses the [log and throw > antipattern|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L406], > which doesn't allow users to ignore certain types of errors without getting > their logs spammed. > The log statement should be removed completely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15710) Update BucketingSinkMigrationTest to restore from 1.10 savepoint
[
https://issues.apache.org/jira/browse/FLINK-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15710:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Update BucketingSinkMigrationTest to restore from 1.10 savepoint
>
>
> Key: FLINK-15710
> URL: https://issues.apache.org/jira/browse/FLINK-15710
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 1.11.0
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Update {{BucketingSinkMigrationTest}} to restore from 1.10 savepoint
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15298) Wrong dependencies in the DataStream API tutorial (the wiki-edits example)
[
https://issues.apache.org/jira/browse/FLINK-15298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15298:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Wrong dependencies in the DataStream API tutorial (the wiki-edits example)
> --
>
> Key: FLINK-15298
> URL: https://issues.apache.org/jira/browse/FLINK-15298
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Documentation
>Affects Versions: 1.7.2, 1.8.3, 1.9.1
>Reporter: Jun Qin
>Assignee: Jun Qin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.3, 1.13.0, 1.11.4
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [The DataStream API Tutorial in Flink 1.9 |
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/getting-started/tutorials/datastream_api.html]
> mentioned the following dependences:
> {code:java}
>
>
> org.apache.flink
> flink-java
> ${flink.version}
>
>
> org.apache.flink
> flink-streaming-java_2.11
> ${flink.version}
>
>
> org.apache.flink
> flink-clients_2.11
> ${flink.version}
>
>
> org.apache.flink
> flink-connector-wikiedits_2.11
> ${flink.version}
>
>
> {code}
> There are two issues here:
> # {{flink-java}} and {{flink-streaming-java}} should be set to *provided*
> scope
> # {{flink-client}} is not needed. If {{flink-client}} is added into *compile*
> scope, {{flink-runtime}} will be added implicitly
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15378:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14998) Remove FileUtils#deletePathIfEmpty
[
https://issues.apache.org/jira/browse/FLINK-14998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-14998:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Remove FileUtils#deletePathIfEmpty
> --
>
> Key: FLINK-14998
> URL: https://issues.apache.org/jira/browse/FLINK-14998
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.13.0
>
>
> With the lesson learned from FLINK-7266, and the refactor of FLINK-8540,
> method of {{FileUtils#deletePathIfEmpty}} has been totally useless in Flink
> production code. From my point of view, it's not wise to provide a method
> with already known high-risk defect in Flink official code. I suggest to
> remove this part of code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15717) Update StatefulJobSavepointMigrationITCase.scala to restore from 1.10 savepoint
[
https://issues.apache.org/jira/browse/FLINK-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15717:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Update StatefulJobSavepointMigrationITCase.scala to restore from 1.10
> savepoint
> ---
>
> Key: FLINK-15717
> URL: https://issues.apache.org/jira/browse/FLINK-15717
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream, Tests
>Affects Versions: 1.11.0
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Update {{StatefulJobSavepointMigrationITCase.scala}} to restore from 1.10
> savepoint
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15012) Checkpoint directory not cleaned up
[
https://issues.apache.org/jira/browse/FLINK-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15012:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Checkpoint directory not cleaned up
> ---
>
> Key: FLINK-15012
> URL: https://issues.apache.org/jira/browse/FLINK-15012
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.1
>Reporter: Nico Kruber
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I started a Flink cluster with 2 TMs using {{start-cluster.sh}} and the
> following config (in addition to the default {{flink-conf.yaml}})
> {code:java}
> state.checkpoints.dir: file:///path/to/checkpoints/
> state.backend: rocksdb {code}
> After submitting a jobwith checkpoints enabled (every 5s), checkpoints show
> up, e.g.
> {code:java}
> bb969f842bbc0ecc3b41b7fbe23b047b/
> ├── chk-2
> │ ├── 238969e1-6949-4b12-98e7-1411c186527c
> │ ├── 2702b226-9cfc-4327-979d-e5508ab2e3d5
> │ ├── 4c51cb24-6f71-4d20-9d4c-65ed6e826949
> │ ├── e706d574-c5b2-467a-8640-1885ca252e80
> │ └── _metadata
> ├── shared
> └── taskowned {code}
> If I shut down the cluster via {{stop-cluster.sh}}, these files will remain
> on disk and not be cleaned up.
> In contrast, if I cancel the job, at least {{chk-2}} will be deleted, but
> still leaving the (empty) directories.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15721) Update StatefulJobWBroadcastStateMigrationITCase to restore from 1.10 savepoint
[
https://issues.apache.org/jira/browse/FLINK-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15721:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Update StatefulJobWBroadcastStateMigrationITCase to restore from 1.10
> savepoint
> ---
>
> Key: FLINK-15721
> URL: https://issues.apache.org/jira/browse/FLINK-15721
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream, Tests
>Affects Versions: 1.11.0
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Update {{StatefulJobWBroadcastStateMigrationITCase}} to restore from 1.10
> savepoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15693) Stop receiving incoming RPC messages when RpcEndpoint is closing
[
https://issues.apache.org/jira/browse/FLINK-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-15693:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Stop receiving incoming RPC messages when RpcEndpoint is closing
>
>
> Key: FLINK-15693
> URL: https://issues.apache.org/jira/browse/FLINK-15693
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.1, 1.10.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.13.0
>
>
> When calling {{RpcEndpoint#closeAsync()}}, the system triggers
> {{RpcEndpoint#onStop}} and transitions the endpoint into the
> {{TerminatingState}}. In order to allow asynchronous clean up operations, the
> main thread executor is not shut down immediately. As a side effect, the
> {{RpcEndpoint}} still accepts incoming RPC messages from other components.
> I think it would be cleaner to no longer accept incoming RPC messages once we
> are in the {{TerminatingState}}. That way we would not worry about the
> internal state of the {{RpcEndpoint}} when processing RPC messages (similar
> to
> [here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java#L952]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
