[GitHub] [flink-web] MartijnVisser commented on a change in pull request #506: [FLINK-25882] Translate updated privacy policy to Chinese
MartijnVisser commented on a change in pull request #506: URL: https://github.com/apache/flink-web/pull/506#discussion_r799235727 ## File path: content/news/2022/01/31/release-statefun-3.2.0.html ## @@ -33,6 +33,8 @@ /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ /* We explicitly disable cookie tracking to avoid privacy issues */ _paq.push(['disableCookies']); + /* Measure a visit to flink.apache.org and nightlies.apache.org/flink as the same visit */ + _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]); Review comment: I was not expecting this to appear in this PR, but it seems that you've found one page where this wasn't properly set yet when you were rebuilding the website. Thanks :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25947) License checker doesn't cover flink-table-planner
[ https://issues.apache.org/jira/browse/FLINK-25947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25947: --- Labels: pull-request-available (was: ) > License checker doesn't cover flink-table-planner > - > > Key: FLINK-25947 > URL: https://issues.apache.org/jira/browse/FLINK-25947 > Project: Flink > Issue Type: Technical Debt > Components: Build System >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18625: [FLINK-25947][build] Simplify shade-plugin configuration
flinkbot commented on pull request #18625: URL: https://github.com/apache/flink/pull/18625#issuecomment-1029739200 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 47f8d63ed2f102d08db7e28e0f935340690599e1 (Fri Feb 04 08:00:49 UTC 2022) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18625: [FLINK-25947][build] Simplify shade-plugin configuration
flinkbot edited a comment on pull request #18625: URL: https://github.com/apache/flink/pull/18625#issuecomment-1029737971 ## CI report: * 47f8d63ed2f102d08db7e28e0f935340690599e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30733) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp merged pull request #18536: [FLINK-25432] Introduces ResourceCleaner to be used in the Dispatcher
XComp merged pull request #18536: URL: https://github.com/apache/flink/pull/18536 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18353: [FLINK-25129][docs]project configuation changes in docs
zentol commented on a change in pull request #18353:
URL: https://github.com/apache/flink/pull/18353#discussion_r799237575
##
File path: docs/content/docs/dev/configuration/overview.md
##
@@ -94,14 +92,12 @@ plugins {
id 'com.github.johnrengelman.shadow' version '7.1.2'
}
// artifact properties
-group = 'org.myorg.quickstart'
+group = 'org.quickstart'
version = '0.1-SNAPSHOT'
-mainClassName = 'org.myorg.quickstart.StreamingJob'
+mainClassName = 'org.quickstart.StreamingJob'
+mainClassName = 'org.quickstart.StreamingJob'
description = """Flink Quickstart Job"""
ext {
-javaVersion = '1.8'
-flinkVersion = '{{< version >}}'
Review comment:
these 2 properties were used though ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18353: [FLINK-25129][docs]project configuation changes in docs
zentol commented on a change in pull request #18353:
URL: https://github.com/apache/flink/pull/18353#discussion_r799237764
##
File path: docs/content/docs/dev/configuration/overview.md
##
@@ -181,22 +171,6 @@ rootProject.name = 'quickstart'
bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{<
version >}} {{< scala_version >}}
```
{{< /tab >}}
-{{< tab "sbt" >}}
-You can scaffold a new Flink project with the following [giter8
template](https://github.com/tillrohrmann/flink-project.g8)
-and the `sbt new` command (which creates new build definitions from a
template) or use the provided quickstart bash script.
-
-### sbt template
-
-```bash
-$ sbt new tillrohrmann/flink-project.g8
Review comment:
Why was this removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #18539: [FLINK-25745] Support RocksDB incremental native savepoints
dawidwys merged pull request #18539: URL: https://github.com/apache/flink/pull/18539 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #18612: [FLINK-25573][kafka] Migrate KafkaSink to SinkV2
fapaul commented on pull request #18612: URL: https://github.com/apache/flink/pull/18612#issuecomment-1029741114 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25745) Support RocksDB incremental native savepoints
[ https://issues.apache.org/jira/browse/FLINK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-25745. Resolution: Implemented Implemented in b54acb63ca1cf93b6f06966775922b623ca6f091..d91cb003221d65e07e135d510ff897f7520add6f > Support RocksDB incremental native savepoints > - > > Key: FLINK-25745 > URL: https://issues.apache.org/jira/browse/FLINK-25745 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Piotr Nowojski >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Respect CheckpointType.SharingFilesStrategy#NO_SHARING flag in > RocksIncrementalSnapshotStrategy. We also need to make sure that > RocksDBIncrementalSnapshotStrategy is creating self contained/relocatable > snapshots (using CheckpointedStateScope#EXCLUSIVE for native savepoints) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] MartijnVisser commented on pull request #18220: [FLINK-25410] Flink CLI should exit when app is accepted with detach …
MartijnVisser commented on pull request #18220: URL: https://github.com/apache/flink/pull/18220#issuecomment-1029741777 @zuston The community is busy due to the upcoming feature freeze for 1.15, which is also one of the reasons why it's probably taking @tillrohrmann a bit longer to respond. Since he's already invested time and energy into this PR, I would recommend to wait for him to finish the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25432) Implement cleanup strategy
[
https://issues.apache.org/jira/browse/FLINK-25432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486865#comment-17486865
]
Matthias Pohl commented on FLINK-25432:
---
{quote}This means that we have to wait for the leadership resources being freed
before cleaning the HA data for the job, still. This is not necessary when
considering the work done by FLINK-24038 which introduces a single leader
election per JobManager.
{quote}
This issue was addressed in the changes. We're still calling the close method
of {{JobManagerRunner}} before cleaning the {{{}HighAvailabilityServices{}}}.
The prioritization can be removed in the context of FLINK-25806. I commented on
that in FLINK-25806 as well.
> Implement cleanup strategy
> --
>
> Key: FLINK-25432
> URL: https://issues.apache.org/jira/browse/FLINK-25432
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We want to combine the job-specific cleanup of the different resources and
> provide a common {{ResourceCleaner}} taking care of the actual cleanup of all
> resources.
> This needs to be integrated into the {{Dispatcher}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] MartijnVisser commented on a change in pull request #18353: [FLINK-25129][docs]project configuation changes in docs
MartijnVisser commented on a change in pull request #18353:
URL: https://github.com/apache/flink/pull/18353#discussion_r799240257
##
File path: docs/content/docs/dev/configuration/overview.md
##
@@ -181,22 +171,6 @@ rootProject.name = 'quickstart'
bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{<
version >}} {{< scala_version >}}
```
{{< /tab >}}
-{{< tab "sbt" >}}
-You can scaffold a new Flink project with the following [giter8
template](https://github.com/tillrohrmann/flink-project.g8)
-and the `sbt new` command (which creates new build definitions from a
template) or use the provided quickstart bash script.
-
-### sbt template
-
-```bash
-$ sbt new tillrohrmann/flink-project.g8
Review comment:
We talked to @tillrohrmann about it - The SBT example was still
targeting 0.13.13 while SBT 1.6.1 is the latest version. We decided to first
remove it and create a follow-up ticket for adding an SBT example later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25432) Implement cleanup strategy
[
https://issues.apache.org/jira/browse/FLINK-25432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486866#comment-17486866
]
Matthias Pohl commented on FLINK-25432:
---
First PR #18536 got merged to {{master}}:
* {{bf30a9b175648fea4dda7aab748a6b6d73dfba27}}
* {{8d27e9ccad70cfd37168e1dec5bbd03386603b3d}}
* {{f96b699d70a27dfad69a3e12afc3b708cc708700}}
* {{b936acb7be50ccc9fed0b32fe4bc0a9185748604}}
* {{a6ef34392638bf724f020071de3d4f118ee753cf}}
* {{6f798996dad0e41ced3e9e293ee81d0caf225874}}
* {{cc5d321d70fa1e0600ea56e5940b07877913ebf8}}
> Implement cleanup strategy
> --
>
> Key: FLINK-25432
> URL: https://issues.apache.org/jira/browse/FLINK-25432
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We want to combine the job-specific cleanup of the different resources and
> provide a common {{ResourceCleaner}} taking care of the actual cleanup of all
> resources.
> This needs to be integrated into the {{Dispatcher}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18612: [FLINK-25573][kafka] Migrate KafkaSink to SinkV2
flinkbot edited a comment on pull request #18612: URL: https://github.com/apache/flink/pull/18612#issuecomment-1028184927 ## CI report: * fc0e9caab54977ae65e9b3005add24f8651fd961 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30715) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25432) Implement cleanup strategy
[
https://issues.apache.org/jira/browse/FLINK-25432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486866#comment-17486866
]
Matthias Pohl edited comment on FLINK-25432 at 2/4/22, 8:07 AM:
First [PR #18536|https://github.com/apache/flink/pull/18536] got merged to
{{master}}:
* {{bf30a9b175648fea4dda7aab748a6b6d73dfba27}}
* {{8d27e9ccad70cfd37168e1dec5bbd03386603b3d}}
* {{f96b699d70a27dfad69a3e12afc3b708cc708700}}
* {{b936acb7be50ccc9fed0b32fe4bc0a9185748604}}
* {{a6ef34392638bf724f020071de3d4f118ee753cf}}
* {{6f798996dad0e41ced3e9e293ee81d0caf225874}}
* {{cc5d321d70fa1e0600ea56e5940b07877913ebf8}}
was (Author: mapohl):
First PR #18536 got merged to {{master}}:
* {{bf30a9b175648fea4dda7aab748a6b6d73dfba27}}
* {{8d27e9ccad70cfd37168e1dec5bbd03386603b3d}}
* {{f96b699d70a27dfad69a3e12afc3b708cc708700}}
* {{b936acb7be50ccc9fed0b32fe4bc0a9185748604}}
* {{a6ef34392638bf724f020071de3d4f118ee753cf}}
* {{6f798996dad0e41ced3e9e293ee81d0caf225874}}
* {{cc5d321d70fa1e0600ea56e5940b07877913ebf8}}
> Implement cleanup strategy
> --
>
> Key: FLINK-25432
> URL: https://issues.apache.org/jira/browse/FLINK-25432
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We want to combine the job-specific cleanup of the different resources and
> provide a common {{ResourceCleaner}} taking care of the actual cleanup of all
> resources.
> This needs to be integrated into the {{Dispatcher}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-25951) Implement Matomo on JavaDocs
[ https://issues.apache.org/jira/browse/FLINK-25951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-25951: -- Assignee: Robert Metzger (was: Martijn Visser) > Implement Matomo on JavaDocs > > > Key: FLINK-25951 > URL: https://issues.apache.org/jira/browse/FLINK-25951 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Martijn Visser >Assignee: Robert Metzger >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18612: [FLINK-25573][kafka] Migrate KafkaSink to SinkV2
flinkbot edited a comment on pull request #18612: URL: https://github.com/apache/flink/pull/18612#issuecomment-1028184927 ## CI report: * fc0e9caab54977ae65e9b3005add24f8651fd961 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30715) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25953) Reorganizes tests around Dispatcher cleanup
Matthias Pohl created FLINK-25953:
-
Summary: Reorganizes tests around Dispatcher cleanup
Key: FLINK-25953
URL: https://issues.apache.org/jira/browse/FLINK-25953
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl
FLINK-25432 introduced new interfaces for the local and global cleanup of
job-related data. This enables us reorganize tests (more specifically
{{DispatcherCleanupResourcesTest}}).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25954) Add cleanup tests to BlobServerCleanupTest
Matthias Pohl created FLINK-25954:
-
Summary: Add cleanup tests to BlobServerCleanupTest
Key: FLINK-25954
URL: https://issues.apache.org/jira/browse/FLINK-25954
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl
FLINK-25432 added cleanup interfaces to the {{BlobServer}} but is still missing
corresponding tests. This should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-25954) Add cleanup tests to BlobServerCleanupTest
[
https://issues.apache.org/jira/browse/FLINK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-25954:
-
Assignee: Matthias Pohl
> Add cleanup tests to BlobServerCleanupTest
> --
>
> Key: FLINK-25954
> URL: https://issues.apache.org/jira/browse/FLINK-25954
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> FLINK-25432 added cleanup interfaces to the {{BlobServer}} but is still
> missing corresponding tests. This should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25955) Add JobManagerRunner implementation that picks up dirty job results to be cleaned up
Matthias Pohl created FLINK-25955:
-
Summary: Add JobManagerRunner implementation that picks up dirty
job results to be cleaned up
Key: FLINK-25955
URL: https://issues.apache.org/jira/browse/FLINK-25955
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl
We want to add a {{JobManagerRunner}} implementation that picks up dirty job
results provided to the {{Dispatcher}}. Instead of rerunning those jobs, only
the cleanup shall be triggered.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-25953) Reorganizes tests around Dispatcher cleanup
[
https://issues.apache.org/jira/browse/FLINK-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-25953:
-
Assignee: Matthias Pohl
> Reorganizes tests around Dispatcher cleanup
> ---
>
> Key: FLINK-25953
> URL: https://issues.apache.org/jira/browse/FLINK-25953
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> FLINK-25432 introduced new interfaces for the local and global cleanup of
> job-related data. This enables us reorganize tests (more specifically
> {{DispatcherCleanupResourcesTest}}).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-25955) Add JobManagerRunner implementation that picks up dirty job results to be cleaned up
[
https://issues.apache.org/jira/browse/FLINK-25955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-25955:
-
Assignee: Matthias Pohl
> Add JobManagerRunner implementation that picks up dirty job results to be
> cleaned up
>
>
> Key: FLINK-25955
> URL: https://issues.apache.org/jira/browse/FLINK-25955
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> We want to add a {{JobManagerRunner}} implementation that picks up dirty job
> results provided to the {{Dispatcher}}. Instead of rerunning those jobs, only
> the cleanup shall be triggered.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25924) KDF Integration tests intermittently fails
[
https://issues.apache.org/jira/browse/FLINK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486859#comment-17486859
]
Chesnay Schepler edited comment on FLINK-25924 at 2/4/22, 8:14 AM:
---
We're experiencing disk space problems which are collected in FLINK-24433 right
now. But the most-recent occurrences happen in
{{{}KinesisFirehoseSinkITCase{}}}. Hence, I'm wondering whether they are
related to this issue. ([example build
failure|https://dev.azure.com/mapohl/flink/_build/results?buildId=680&view=logs&j=dafbab6d-4616-5d7b-ee37-3c54e4828fd7&t=e204f081-e6cd-5c04-4f4c-919639b63be9&l=1001])
was (Author: mapohl):
We're experiencing memory problems which are collected in FLINK-24433 right
now. But the most-recent occurrences happen in
{{{}KinesisFirehoseSinkITCase{}}}. Hence, I'm wondering whether they are
related to this issue. ([example build
failure|https://dev.azure.com/mapohl/flink/_build/results?buildId=680&view=logs&j=dafbab6d-4616-5d7b-ee37-3c54e4828fd7&t=e204f081-e6cd-5c04-4f4c-919639b63be9&l=1001])
> KDF Integration tests intermittently fails
> --
>
> Key: FLINK-25924
> URL: https://issues.apache.org/jira/browse/FLINK-25924
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.0
>Reporter: Zichen Liu
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> Intermittent failures introduced as part of merge (PR#18314:
> [FLINK-24228[connectors/firehose] - Unified Async Sink for Kinesis
> Firehose|https://github.com/apache/flink/pull/18314]):
> # Failures are intermittent and affecting c. 1 in 7 of builds- on
> {{flink-ci.flink}} and {{flink-ci.flink-master-mirror}} .
> # The issue looks identical to the KinesaliteContainer startup issue
> (Appendix 1).
> # I have managed to reproduce the issue locally - if I start some parallel
> containers and keep them running - and then run {{KinesisFirehoseSinkITCase}}
> then c. 1 in 6 gives the error.
> # The errors have a slightly different appearance on
> {{flink-ci.flink-master-mirror}} vs {{flink-ci.flink}} which has the same
> appearance as local. I only hope it is a difference in logging/killing
> environment variables. (and that there aren’t 2 distinct issues)
> Appendix 1:
> {code:java}
> org.testcontainers.containers.ContainerLaunchException: Container startup
> failed
> at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
> at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
> at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1066)
> at
> ... 11 more
> Caused by: org.testcontainers.containers.ContainerLaunchException: Could not
> create/start container
> at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
> at
> org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
> at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
> ... 12 more
> Caused by: org.rnorth.ducttape.TimeoutException: Timeout waiting for result
> with exception
> at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:54)
> at
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24433) "No space left on device" in Azure e2e tests
[
https://issues.apache.org/jira/browse/FLINK-24433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486869#comment-17486869
]
Chesnay Schepler commented on FLINK-24433:
--
I'm inclined to disable the test.
> "No space left on device" in Azure e2e tests
>
>
> Key: FLINK-24433
> URL: https://issues.apache.org/jira/browse/FLINK-24433
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.15.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24668&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=070ff179-953e-5bda-71fa-d6599415701c&l=19772
> {code}
> Sep 30 17:08:42 Job has been submitted with JobID
> 5594c18e128a328ede39cfa59cb3cb07
> Sep 30 17:08:56 2021-09-30 17:08:56,809 main ERROR Recovering from
> StringBuilderEncoder.encode('2021-09-30 17:08:56,807 WARN
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - An
> exception occurred when fetching query results
> Sep 30 17:08:56 java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., Sep 30 17:08:56 org.apache.flink.runtime.messages.FlinkJobNotFoundException:
> Could not find Flink job (5594c18e128a328ede39cfa59cb3cb07)
> Sep 30 17:08:56 at
> org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGateway(Dispatcher.java:923)
> Sep 30 17:08:56 at
> org.apache.flink.runtime.dispatcher.Dispatcher.performOperationOnJobMasterGateway(Dispatcher.java:937)
> Sep 30 17:08:56 at
> org.apache.flink.runtime.dispatcher.Dispatcher.deliverCoordinationRequestToCoordina2021-09-30T17:08:57.1584224Z
> ##[error]No space left on device
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] XComp opened a new pull request #18626: [FLINK-25953][FLINK-25954] Updates and reorganizes tests around Dispatcher cleanup
XComp opened a new pull request #18626: URL: https://github.com/apache/flink/pull/18626 ## What is the purpose of the change Refactored the tests around cleanup collecting cleanup-relevant tests in `DispatcherResourceCleanupTest` ## Brief change log * Makes `BlobServerClenupTest` based on JUnit5 and AssertJ and adds cleanup-related tests * `DispatcherResourceCleanupTest` is refactored to only rely on the cleanup without testing the downstream CleanableResources. Testing that the right components are cleaned is tested in `DispatcherResourceCleanerFactoryTest`. The actual cleanup for each component is then tested in the corresponding component tests. * Additionally, I added test cases in `DispatcherResourceCleanerFactoryTest` for verifying that the `JobResult` is marked as dirty before the cleanup and clean after the cleanup ## Verifying this change The PR itself is just tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25953) Reorganizes tests around Dispatcher cleanup
[
https://issues.apache.org/jira/browse/FLINK-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25953:
---
Labels: pull-request-available (was: )
> Reorganizes tests around Dispatcher cleanup
> ---
>
> Key: FLINK-25953
> URL: https://issues.apache.org/jira/browse/FLINK-25953
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> FLINK-25432 introduced new interfaces for the local and global cleanup of
> job-related data. This enables us reorganize tests (more specifically
> {{DispatcherCleanupResourcesTest}}).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18626: [FLINK-25953][FLINK-25954] Updates and reorganizes tests around Dispatcher cleanup
flinkbot commented on pull request #18626: URL: https://github.com/apache/flink/pull/18626#issuecomment-1029753226 ## CI report: * cd117b6fbf03cfdeff52f11129ccf2f6d232ad69 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
echauchot commented on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1029754467 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
flinkbot edited a comment on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1028011200 ## CI report: * Unknown: [CANCELED](TBD) * 326a658958b6dc91c90ec25f0f501a4947a00c17 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18626: [FLINK-25953][FLINK-25954] Updates and reorganizes tests around Dispatcher cleanup
flinkbot edited a comment on pull request #18626: URL: https://github.com/apache/flink/pull/18626#issuecomment-1029753226 ## CI report: * cd117b6fbf03cfdeff52f11129ccf2f6d232ad69 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30735) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18353: [FLINK-25129][docs]project configuation changes in docs
infoverload commented on a change in pull request #18353:
URL: https://github.com/apache/flink/pull/18353#discussion_r799252625
##
File path: docs/content/docs/dev/configuration/overview.md
##
@@ -181,22 +171,6 @@ rootProject.name = 'quickstart'
bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- {{<
version >}} {{< scala_version >}}
```
{{< /tab >}}
-{{< tab "sbt" >}}
-You can scaffold a new Flink project with the following [giter8
template](https://github.com/tillrohrmann/flink-project.g8)
-and the `sbt new` command (which creates new build definitions from a
template) or use the provided quickstart bash script.
-
-### sbt template
-
-```bash
-$ sbt new tillrohrmann/flink-project.g8
Review comment:
I had already removed the standalone page on SBT and had forgotten to
remove this section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18626: [FLINK-25953][FLINK-25954] Updates and reorganizes tests around Dispatcher cleanup
flinkbot commented on pull request #18626: URL: https://github.com/apache/flink/pull/18626#issuecomment-1029756008 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cd117b6fbf03cfdeff52f11129ccf2f6d232ad69 (Fri Feb 04 08:28:15 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-25432) Implement cleanup strategy
[
https://issues.apache.org/jira/browse/FLINK-25432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-25432.
---
Fix Version/s: 1.15.0
Resolution: Fixed
> Implement cleanup strategy
> --
>
> Key: FLINK-25432
> URL: https://issues.apache.org/jira/browse/FLINK-25432
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> We want to combine the job-specific cleanup of the different resources and
> provide a common {{ResourceCleaner}} taking care of the actual cleanup of all
> resources.
> This needs to be integrated into the {{Dispatcher}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25432) Introduce common interfaces for cleaning up local and global job data
[
https://issues.apache.org/jira/browse/FLINK-25432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-25432:
--
Summary: Introduce common interfaces for cleaning up local and global job
data (was: Implement cleanup strategy)
> Introduce common interfaces for cleaning up local and global job data
> -
>
> Key: FLINK-25432
> URL: https://issues.apache.org/jira/browse/FLINK-25432
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> We want to combine the job-specific cleanup of the different resources and
> provide a common {{ResourceCleaner}} taking care of the actual cleanup of all
> resources.
> This needs to be integrated into the {{Dispatcher}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18353: [FLINK-25129][docs]project configuation changes in docs
flinkbot edited a comment on pull request #18353: URL: https://github.com/apache/flink/pull/18353#issuecomment-1012981377 ## CI report: * ddabc7bb39a84b79407d5c9b85de9c83d0959de2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30708) * 059b1a0df499474c5f672df131abc826660101a8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
flinkbot edited a comment on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1028011200 ## CI report: * Unknown: [CANCELED](TBD) * 326a658958b6dc91c90ec25f0f501a4947a00c17 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30736) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18353: [FLINK-25129][docs]project configuation changes in docs
flinkbot edited a comment on pull request #18353: URL: https://github.com/apache/flink/pull/18353#issuecomment-1012981377 ## CI report: * ddabc7bb39a84b79407d5c9b85de9c83d0959de2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30708) * 059b1a0df499474c5f672df131abc826660101a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30737) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24229) [FLIP-171] DynamoDB implementation of Async Sink
[
https://issues.apache.org/jira/browse/FLINK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486873#comment-17486873
]
Danny Cranmer commented on FLINK-24229:
---
[~Gusev] I plan to start on this review next week. I will do my best to get it
in before the 1.15 feature freeze
> [FLIP-171] DynamoDB implementation of Async Sink
>
>
> Key: FLINK-24229
> URL: https://issues.apache.org/jira/browse/FLINK-24229
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
>Reporter: Zichen Liu
>Assignee: Zichen Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> h2. Motivation
> *User stories:*
> As a Flink user, I’d like to use DynamoDB as sink for my data pipeline.
> *Scope:*
> * Implement an asynchronous sink for DynamoDB by inheriting the
> AsyncSinkBase class. The implementation can for now reside in its own module
> in flink-connectors.
> * Implement an asynchornous sink writer for DynamoDB by extending the
> AsyncSinkWriter. The implementation must deal with failed requests and retry
> them using the {{requeueFailedRequestEntry}} method. If possible, the
> implementation should batch multiple requests (PutRecordsRequestEntry
> objects) to Firehose for increased throughput. The implemented Sink Writer
> will be used by the Sink class that will be created as part of this story.
> * Java / code-level docs.
> * End to end testing: add tests that hits a real AWS instance. (How to best
> donate resources to the Flink project to allow this to happen?)
> h2. References
> More details to be found
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25943) New Kinesis, Firehose to provide a state serializer
[
https://issues.apache.org/jira/browse/FLINK-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul updated FLINK-25943:
Affects Version/s: 1.15.0
> New Kinesis, Firehose to provide a state serializer
> ---
>
> Key: FLINK-25943
> URL: https://issues.apache.org/jira/browse/FLINK-25943
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.15.0
>Reporter: Zichen Liu
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> AsyncSinkWriter implements snapshotState to write the pending request into
> state but none of the implementation (Kinesis, Firehose) provides a state
> serializer nor interacts with the recovered state.
>
> * Implement
> {code:java}
> getWriterStateSerializer{code}
> for the Kinesis/Firehose Sinks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25943) New Kinesis, Firehose to provide a state serializer
[
https://issues.apache.org/jira/browse/FLINK-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul updated FLINK-25943:
Priority: Critical (was: Major)
> New Kinesis, Firehose to provide a state serializer
> ---
>
> Key: FLINK-25943
> URL: https://issues.apache.org/jira/browse/FLINK-25943
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Kinesis
>Reporter: Zichen Liu
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> AsyncSinkWriter implements snapshotState to write the pending request into
> state but none of the implementation (Kinesis, Firehose) provides a state
> serializer nor interacts with the recovered state.
>
> * Implement
> {code:java}
> getWriterStateSerializer{code}
> for the Kinesis/Firehose Sinks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25943) New Kinesis, Firehose to provide a state serializer
[
https://issues.apache.org/jira/browse/FLINK-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul updated FLINK-25943:
Issue Type: Bug (was: New Feature)
> New Kinesis, Firehose to provide a state serializer
> ---
>
> Key: FLINK-25943
> URL: https://issues.apache.org/jira/browse/FLINK-25943
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Reporter: Zichen Liu
>Assignee: Ahmed Hamdy
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> AsyncSinkWriter implements snapshotState to write the pending request into
> state but none of the implementation (Kinesis, Firehose) provides a state
> serializer nor interacts with the recovered state.
>
> * Implement
> {code:java}
> getWriterStateSerializer{code}
> for the Kinesis/Firehose Sinks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] XComp opened a new pull request #18627: [FLINK-25955][runtime] Introduces `JobManagerRunner` implementation for cleanup
XComp opened a new pull request #18627: URL: https://github.com/apache/flink/pull/18627 This PR is related to [FLIP-194](https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore) and based on PR #18626 . It's labeled as a draft as long as PR #18626 isn't merged into `master`. ## What is the purpose of the change This change introduces a `JobManagerRunner` implementation taking care of the cleanup of a otherwise finished job. ## Brief change log * makes certain `Testing*` implementations more general to make testing of the new functionality easier * Moves extraction of `state.checkpoints.num-retained` from `SchedulerUtils` into `DefaultCompletedCheckpointStoreUtils` (which is renamed into `CompletedCheckpointStoreUtils`) to align with the new use case for this functionality * Makes `ApplicationStatus` <-> `JobStatus` mapping (partially) symmetric * Introduces `CheckpointResourcesCleanupRunner` as a new `JobManagerRunner` implementation (+ correspondng factory class to support testing) and integrates it into the `Dispatcher` * Reorganizes Dispatcher tests moving certain tests out of `DispatcherTest` into `DispatcherResourceCleanupTest` ## Verifying this change This change added tests and can be verified as follows: * ApplicationMode is tests in `ApplicationDispatcheerBootstrapITCase` * General Dispatcher is tested in `DispatcherFailoverITCase` * `DispatcherTest.testJobCleanupWithoutRecoveredJobGraph` testing that the job is cleaned up through a `CheckpointResourcesCleanupRunner` rather than initializing a new `JobMaster` * `CheckpointResourcesCleanupRunnerTest` was added to cover new class `CheckpointResourcesCleanupRunner` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs / JavaDocs * JavaDoc is provided * A `JobResultStore` section is added under `Deployment / High Availability` * `JobResultStore` is added to the glossary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25956) KryoSerializer is noisy in scala-free Flink
Chesnay Schepler created FLINK-25956:
Summary: KryoSerializer is noisy in scala-free Flink
Key: FLINK-25956
URL: https://issues.apache.org/jira/browse/FLINK-25956
Project: Flink
Issue Type: Technical Debt
Components: API / Type Serialization System
Affects Versions: 1.15.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.15.0
When scala is not on the classpath and Kryo is used the {{KryoSerializer}} logs
a full ClassNotFoundException as a warning.
Since this is now an expected situation it should logged on info. The exception
should only be logged if debug logging is enabled.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25955) Add JobManagerRunner implementation that picks up dirty job results to be cleaned up
[
https://issues.apache.org/jira/browse/FLINK-25955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25955:
---
Labels: pull-request-available (was: )
> Add JobManagerRunner implementation that picks up dirty job results to be
> cleaned up
>
>
> Key: FLINK-25955
> URL: https://issues.apache.org/jira/browse/FLINK-25955
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available
>
> We want to add a {{JobManagerRunner}} implementation that picks up dirty job
> results provided to the {{Dispatcher}}. Instead of rerunning those jobs, only
> the cleanup shall be triggered.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zentol opened a new pull request #18628: [FLINK-25956][kryo] Remove noisy warnings
zentol opened a new pull request #18628: URL: https://github.com/apache/flink/pull/18628 When scala is not on the classpath and Kryo is used the KryoSerializer logs a full ClassNotFoundException as a warning. Since this is now an expected situation it should logged on info. The exception should only be logged if debug logging is enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18627: [FLINK-25955][runtime] Introduces `JobManagerRunner` implementation for cleanup
flinkbot commented on pull request #18627: URL: https://github.com/apache/flink/pull/18627#issuecomment-1029768577 ## CI report: * ada8a7c27fc0bb9c023c6df8904e947616132319 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25956) KryoSerializer is noisy in scala-free Flink
[
https://issues.apache.org/jira/browse/FLINK-25956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25956:
---
Labels: pull-request-available (was: )
> KryoSerializer is noisy in scala-free Flink
> ---
>
> Key: FLINK-25956
> URL: https://issues.apache.org/jira/browse/FLINK-25956
> Project: Flink
> Issue Type: Technical Debt
> Components: API / Type Serialization System
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> When scala is not on the classpath and Kryo is used the {{KryoSerializer}}
> logs a full ClassNotFoundException as a warning.
> Since this is now an expected situation it should logged on info. The
> exception should only be logged if debug logging is enabled.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18627: [FLINK-25955][runtime] Introduces `JobManagerRunner` implementation for cleanup
flinkbot commented on pull request #18627: URL: https://github.com/apache/flink/pull/18627#issuecomment-1029768794 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ada8a7c27fc0bb9c023c6df8904e947616132319 (Fri Feb 04 08:49:02 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18628: [FLINK-25956][kryo] Remove noisy warnings
flinkbot commented on pull request #18628: URL: https://github.com/apache/flink/pull/18628#issuecomment-1029770715 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 28e3908299c9460e05000b2b67f17612363ba15d (Fri Feb 04 08:52:07 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18612: [FLINK-25573][kafka] Migrate KafkaSink to SinkV2
flinkbot edited a comment on pull request #18612: URL: https://github.com/apache/flink/pull/18612#issuecomment-1028184927 ## CI report: * fc0e9caab54977ae65e9b3005add24f8651fd961 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30715) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #18353: [FLINK-25129][docs]project configuation changes in docs
zentol merged pull request #18353: URL: https://github.com/apache/flink/pull/18353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18627: [FLINK-25955][runtime] Introduces `JobManagerRunner` implementation for cleanup
flinkbot edited a comment on pull request #18627: URL: https://github.com/apache/flink/pull/18627#issuecomment-1029768577 ## CI report: * ada8a7c27fc0bb9c023c6df8904e947616132319 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30738) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18628: [FLINK-25956][kryo] Remove noisy warnings
flinkbot commented on pull request #18628: URL: https://github.com/apache/flink/pull/18628#issuecomment-1029771230 ## CI report: * 28e3908299c9460e05000b2b67f17612363ba15d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25129) Update docs to use flink-table-planner-loader instead of flink-table-planner
[ https://issues.apache.org/jira/browse/FLINK-25129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-25129. Fix Version/s: 1.15.0 Resolution: Fixed master: d79d08362b1f0032f12afaa0a73c9c035b7faebf > Update docs to use flink-table-planner-loader instead of flink-table-planner > > > Key: FLINK-25129 > URL: https://issues.apache.org/jira/browse/FLINK-25129 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Examples, Table SQL / API >Reporter: Francesco Guardiani >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > For more details > https://docs.google.com/document/d/12yDUCnvcwU2mODBKTHQ1xhfOq1ujYUrXltiN_rbhT34/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18628: [FLINK-25956][kryo] Remove noisy warnings
flinkbot edited a comment on pull request #18628: URL: https://github.com/apache/flink/pull/18628#issuecomment-1029771230 ## CI report: * 28e3908299c9460e05000b2b67f17612363ba15d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30739) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] curcur merged pull request #506: [FLINK-25882] Translate updated privacy policy to Chinese
curcur merged pull request #506: URL: https://github.com/apache/flink-web/pull/506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun-playground] igalshilman commented on a change in pull request #17: Fix the Kubernetes example
igalshilman commented on a change in pull request #17: URL: https://github.com/apache/flink-statefun-playground/pull/17#discussion_r799281747 ## File path: deployments/k8s/04-statefun/00-module.yaml ## @@ -5,11 +5,12 @@ metadata: name: module-config data: module.yaml: |+ -kind: io.statefun.endpoints.v2.http +kind: io.statefun.endpoints.v2/http spec: functions: example/* urlPathTemplate: http://functions.statefun.svc.cluster.local:8000/statefun transport: +type: io.statefun.transports.v1/okhttp Review comment: cool, thanks for doing is! If you need any assistance/stumble something let us know on the user mailing list for Apache Flink. I'd be happy to help out! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun-playground] igalshilman commented on a change in pull request #17: Fix the Kubernetes example
igalshilman commented on a change in pull request #17: URL: https://github.com/apache/flink-statefun-playground/pull/17#discussion_r799281747 ## File path: deployments/k8s/04-statefun/00-module.yaml ## @@ -5,11 +5,12 @@ metadata: name: module-config data: module.yaml: |+ -kind: io.statefun.endpoints.v2.http +kind: io.statefun.endpoints.v2/http spec: functions: example/* urlPathTemplate: http://functions.statefun.svc.cluster.local:8000/statefun transport: +type: io.statefun.transports.v1/okhttp Review comment: cool, thanks for doing is! If you need any assistance/stumble upon something let us know on the user mailing list for Apache Flink. I'd be happy to help out! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-25882) Translate updated privacy policy to Chinese
[
https://issues.apache.org/jira/browse/FLINK-25882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser resolved FLINK-25882.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master: 03ea9fe0181c2f2bf469f093e5553079eb76d1de
> Translate updated privacy policy to Chinese
> ---
>
> Key: FLINK-25882
> URL: https://issues.apache.org/jira/browse/FLINK-25882
> Project: Flink
> Issue Type: Sub-task
> Components: Project Website
>Reporter: Martijn Visser
>Assignee: Yubin Li
>Priority: Major
> Labels: chinese-translation, pull-request-available
> Fix For: 1.15.0
>
>
> After https://github.com/apache/flink-web/pull/503 is merged, it requires
> {{privacy-policy.zh.md}} to be translated to Chinese.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25957) Cache cassandra/pulsar images
Chesnay Schepler created FLINK-25957: Summary: Cache cassandra/pulsar images Key: FLINK-25957 URL: https://issues.apache.org/jira/browse/FLINK-25957 Project: Flink Issue Type: Technical Debt Components: Test Infrastructure Affects Versions: 1.15.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25957) Cache cassandra/pulsar images
[ https://issues.apache.org/jira/browse/FLINK-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-25957: - Component/s: Build System / Azure Pipelines (was: Test Infrastructure) > Cache cassandra/pulsar images > - > > Key: FLINK-25957 > URL: https://issues.apache.org/jira/browse/FLINK-25957 > Project: Flink > Issue Type: Technical Debt > Components: Build System / Azure Pipelines >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18629: [FLINK-25957][ci] Cache cassandra/pulsar images
flinkbot commented on pull request #18629: URL: https://github.com/apache/flink/pull/18629#issuecomment-1029791726 ## CI report: * 49be6965d12a87be774f23f6572be47e1052ce28 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25957) Cache cassandra/pulsar images
[ https://issues.apache.org/jira/browse/FLINK-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25957: --- Labels: pull-request-available (was: ) > Cache cassandra/pulsar images > - > > Key: FLINK-25957 > URL: https://issues.apache.org/jira/browse/FLINK-25957 > Project: Flink > Issue Type: Technical Debt > Components: Build System / Azure Pipelines >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18629: [FLINK-25957][ci] Cache cassandra/pulsar images
flinkbot commented on pull request #18629: URL: https://github.com/apache/flink/pull/18629#issuecomment-1029794184 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 205839db714f45a5c183f1c73ae9b3c09be06ef6 (Fri Feb 04 09:22:39 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25954) Add cleanup tests to BlobServerCleanupTest
[
https://issues.apache.org/jira/browse/FLINK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486900#comment-17486900
]
Matthias Pohl commented on FLINK-25954:
---
I manually added the PR because the relevant commit is added to the PR fixed
FLINK-25953
> Add cleanup tests to BlobServerCleanupTest
> --
>
> Key: FLINK-25954
> URL: https://issues.apache.org/jira/browse/FLINK-25954
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> FLINK-25432 added cleanup interfaces to the {{BlobServer}} but is still
> missing corresponding tests. This should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18629: [FLINK-25957][ci] Cache cassandra/pulsar images
flinkbot edited a comment on pull request #18629: URL: https://github.com/apache/flink/pull/18629#issuecomment-1029791726 ## CI report: * 49be6965d12a87be774f23f6572be47e1052ce28 UNKNOWN * 205839db714f45a5c183f1c73ae9b3c09be06ef6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys opened a new pull request #18630: [FLINK-25952] Savepoints on S3 are not relocatable even if entropy injection is not enabled
dawidwys opened a new pull request #18630: URL: https://github.com/apache/flink/pull/18630 ## What is the purpose of the change We treat all filesystems that extend from EntropyInjectingFileSystem as if they always inject entropy. However, we support returning null from EntropyInjectingFileSystem#getEntropyInjectionKey which translates to disabled entropy injections. In such cases we should support savepoints relocation by creating relative paths for exclusive files. ## Verifying this change Added tests in: * `FsCheckpointStreamFactoryTest` * `EntropyInjectorTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (**yes** / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25952) Savepoint on S3 are not relocatable even if entropy injection is not enabled
[
https://issues.apache.org/jira/browse/FLINK-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25952:
---
Labels: pull-request-available (was: )
> Savepoint on S3 are not relocatable even if entropy injection is not enabled
>
>
> Key: FLINK-25952
> URL: https://issues.apache.org/jira/browse/FLINK-25952
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Runtime / Checkpointing
>Affects Versions: 1.15.0, 1.13.5, 1.14.3
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.4, 1.13.7
>
>
> We have a limitation that if we create savepoints with an injected entropy,
> they are not relocatable
> (https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/#triggering-savepoints).
> However the check if we use the entropy is flawed. In
> {{FsCheckpointStreamFactory}} we check only if the used filesystem extends
> from {{EntropyInjectingFileSystem}}. {{FlinkS3FileSystem}} does, but it still
> may have the entropy disabled. {{FlinkS3FileSystem#getEntropyInjectionKey}}
> may still return {{null}}. We should check for that in
> {{org.apache.flink.core.fs.EntropyInjector#isEntropyInjecting}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18629: [FLINK-25957][ci] Cache cassandra/pulsar images
flinkbot edited a comment on pull request #18629: URL: https://github.com/apache/flink/pull/18629#issuecomment-1029791726 ## CI report: * 49be6965d12a87be774f23f6572be47e1052ce28 UNKNOWN * 205839db714f45a5c183f1c73ae9b3c09be06ef6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30741) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18630: [FLINK-25952] Savepoints on S3 are not relocatable even if entropy injection is not enabled
flinkbot commented on pull request #18630: URL: https://github.com/apache/flink/pull/18630#issuecomment-1029802936 ## CI report: * 62e647355463a7659bec72d746982e939d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18630: [FLINK-25952] Savepoints on S3 are not relocatable even if entropy injection is not enabled
flinkbot commented on pull request #18630: URL: https://github.com/apache/flink/pull/18630#issuecomment-1029803373 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 62e647355463a7659bec72d746982e939d82 (Fri Feb 04 09:34:53 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25433) Integrate retry strategy for cleanup stage
[
https://issues.apache.org/jira/browse/FLINK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl reassigned FLINK-25433:
-
Assignee: Matthias Pohl
> Integrate retry strategy for cleanup stage
> --
>
> Key: FLINK-25433
> URL: https://issues.apache.org/jira/browse/FLINK-25433
> Project: Flink
> Issue Type: Sub-task
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
>
> The {{ResourceCleaner}} should be able to cleanup not only once but retry
> infinitely.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18630: [FLINK-25952] Savepoints on S3 are not relocatable even if entropy injection is not enabled
flinkbot edited a comment on pull request #18630: URL: https://github.com/apache/flink/pull/18630#issuecomment-1029802936 ## CI report: * 62e647355463a7659bec72d746982e939d82 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30742) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nirtsruya commented on a change in pull request #18518: [FLINK-24229][connectors/dynamodb] Added DynamoDB connector
nirtsruya commented on a change in pull request #18518:
URL: https://github.com/apache/flink/pull/18518#discussion_r799306670
##
File path:
flink-connectors/flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import
org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream
using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to
communicate with the AWS
+ * endpoint.
+ *
+ * The behaviour of the buffering may be specified by providing
configuration during the sink
+ * build time.
+ *
+ *
+ * {@code maxBatchSize}: the maximum size of a batch of entries that may
be written to
+ * DynamoDb.
+ * {@code maxInFlightRequests}: the maximum number of in flight requests
that may exist, if
+ * any more in flight requests need to be initiated once the maximum has
been reached, then it
+ * will be blocked until some have completed
+ * {@code maxBufferedRequests}: the maximum number of elements held in
the buffer, requests to
+ * add elements will be blocked while the number of elements in the
buffer is at the maximum
+ * {@code maxBatchSizeInBytes}: the maximum size of a batch of entries
that may be written to
+ * DynamoDb measured in bytes
+ * {@code maxTimeInBufferMS}: the maximum amount of time an entry is
allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be
flushed immediately
+ * {@code maxRecordSizeInBytes}: the maximum size of a record the sink
will accept into the
+ * buffer, a record of size larger than this will be rejected when
passed to the sink
+ * {@code failOnError}: when an exception is encountered while
persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * {@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb
sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in
the same batch request.
+ * Only the latest record with the same combination of key attributes is
preserved in the
+ * request.
+ *
+ *
+ * Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink extends AsyncSinkBase {
+
+private final Properties dynamoDbClientProperties;
+private final DynamoDbTablesConfig dynamoDbTablesConfig;
+private final boolean failOnError;
+
+protected DynamoDbSink(
+ElementConverter elementConverter,
+int maxBatchSize,
+int maxInFlightRequests,
+int maxBufferedRequests,
+long maxBatchSizeInBytes,
+long maxTimeInBufferMS,
+long maxRecordSizeInBytes,
+boolean failOnError,
+DynamoDbTablesConfig dynamoDbTablesConfig,
+Properties dynamoDbClientProperties) {
+super(
+elementConverter,
+maxBatchSize,
+maxInFlightRequests,
+maxBufferedRequests,
+maxBatchSizeInBytes,
+maxTimeInBufferMS,
+maxRecordSizeInBytes);
+checkNotNull(dynamoDbTablesConfig, "Missing dynamoDbTablesConfig");
+checkNotNull(dynamoDbClientPrope
[GitHub] [flink] nirtsruya commented on a change in pull request #18518: [FLINK-24229][connectors/dynamodb] Added DynamoDB connector
nirtsruya commented on a change in pull request #18518:
URL: https://github.com/apache/flink/pull/18518#discussion_r799306670
##
File path:
flink-connectors/flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import
org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream
using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to
communicate with the AWS
+ * endpoint.
+ *
+ * The behaviour of the buffering may be specified by providing
configuration during the sink
+ * build time.
+ *
+ *
+ * {@code maxBatchSize}: the maximum size of a batch of entries that may
be written to
+ * DynamoDb.
+ * {@code maxInFlightRequests}: the maximum number of in flight requests
that may exist, if
+ * any more in flight requests need to be initiated once the maximum has
been reached, then it
+ * will be blocked until some have completed
+ * {@code maxBufferedRequests}: the maximum number of elements held in
the buffer, requests to
+ * add elements will be blocked while the number of elements in the
buffer is at the maximum
+ * {@code maxBatchSizeInBytes}: the maximum size of a batch of entries
that may be written to
+ * DynamoDb measured in bytes
+ * {@code maxTimeInBufferMS}: the maximum amount of time an entry is
allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be
flushed immediately
+ * {@code maxRecordSizeInBytes}: the maximum size of a record the sink
will accept into the
+ * buffer, a record of size larger than this will be rejected when
passed to the sink
+ * {@code failOnError}: when an exception is encountered while
persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * {@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb
sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in
the same batch request.
+ * Only the latest record with the same combination of key attributes is
preserved in the
+ * request.
+ *
+ *
+ * Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink extends AsyncSinkBase {
+
+private final Properties dynamoDbClientProperties;
+private final DynamoDbTablesConfig dynamoDbTablesConfig;
+private final boolean failOnError;
+
+protected DynamoDbSink(
+ElementConverter elementConverter,
+int maxBatchSize,
+int maxInFlightRequests,
+int maxBufferedRequests,
+long maxBatchSizeInBytes,
+long maxTimeInBufferMS,
+long maxRecordSizeInBytes,
+boolean failOnError,
+DynamoDbTablesConfig dynamoDbTablesConfig,
+Properties dynamoDbClientProperties) {
+super(
+elementConverter,
+maxBatchSize,
+maxInFlightRequests,
+maxBufferedRequests,
+maxBatchSizeInBytes,
+maxTimeInBufferMS,
+maxRecordSizeInBytes);
+checkNotNull(dynamoDbTablesConfig, "Missing dynamoDbTablesConfig");
+checkNotNull(dynamoDbClientPrope
[GitHub] [flink] matriv commented on a change in pull request #18611: [FLINK-24385][table] Introduce TRY_CAST
matriv commented on a change in pull request #18611:
URL: https://github.com/apache/flink/pull/18611#discussion_r798523671
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/BaseExpressions.java
##
@@ -476,6 +477,15 @@ public OutType cast(DataType toType) {
return toApiSpecificExpression(unresolvedCall(CAST, toExpr(),
typeLiteral(toType)));
}
+/**
+ * Converts a value to a given data type.
+ *
Review comment:
Please add the non-failure but `null` return vs normal `cast`
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/core/testutils/FlinkAssertions.java
##
@@ -117,7 +117,7 @@ private FlinkAssertions() {}
* .hasMessageContaining(containsMessage));
* }
*/
-public static ThrowingConsumer anyCauseMatches(String
containsMessage) {
+public static ThrowingConsumer anyCauseMatches(String
containsMessage) {
Review comment:
why `super` ?
##
File path: docs/data/sql_functions.yml
##
@@ -561,7 +561,10 @@ conditional:
conversion:
- sql: CAST(value AS type)
table: ANY.cast(TYPE)
-description: Returns a new value being cast to type type. E.g., CAST('42'
AS INT) returns 42; CAST(NULL AS VARCHAR) returns NULL of type VARCHAR.
+description: Returns a new value being cast to type type. A CAST error
throws an exception and fails the job. If you're performing a cast operation
that may fail, like INT to STRING, you should rather use TRY_CAST, in order to
handle errors. E.g., CAST('42' AS INT) returns 42; CAST(NULL AS VARCHAR)
returns NULL of type VARCHAR; TRY_CAST('non-number' AS INT) throws an exception
and fails the job.
Review comment:
I think we need to refer to the LEGACY BEHAVIOUR config param in
`execution_config_configuration.html` to be 100% clear to the user.
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/BuiltInFunctionTestBase.java
##
@@ -144,58 +143,69 @@ private static void testResult(
final TableResult result = resultTable.execute();
final Iterator iterator = result.collect();
-assertTrue(iterator.hasNext());
+assertThat(iterator).hasNext();
final Row row = iterator.next();
-assertFalse("No more rows expected.", iterator.hasNext());
+assertThat(iterator).as("No more rows expected.").isExhausted();
for (int i = 0; i < row.getArity(); i++) {
-assertEquals(
-"Logical type for spec [" + i + "] of test [" + testItem +
"] doesn't match.",
-expectedDataTypes.get(i).getLogicalType(),
-
result.getResolvedSchema().getColumnDataTypes().get(i).getLogicalType());
-
-assertEquals(
-"Result for spec [" + i + "] of test [" + testItem + "]
doesn't match.",
-// Use Row.equals() to enable equality for complex
structure, i.e. byte[]
-Row.of(testItem.results.get(i)),
-Row.of(row.getField(i)));
+
assertThat(result.getResolvedSchema().getColumnDataTypes().get(i).getLogicalType())
+.as(
+"Logical type for spec ["
Review comment:
nit: I would try to compact this to less line breaks.
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/BuiltInFunctionTestBase.java
##
@@ -144,58 +143,69 @@ private static void testResult(
final TableResult result = resultTable.execute();
final Iterator iterator = result.collect();
-assertTrue(iterator.hasNext());
+assertThat(iterator).hasNext();
final Row row = iterator.next();
-assertFalse("No more rows expected.", iterator.hasNext());
+assertThat(iterator).as("No more rows expected.").isExhausted();
for (int i = 0; i < row.getArity(); i++) {
-assertEquals(
-"Logical type for spec [" + i + "] of test [" + testItem +
"] doesn't match.",
-expectedDataTypes.get(i).getLogicalType(),
-
result.getResolvedSchema().getColumnDataTypes().get(i).getLogicalType());
-
-assertEquals(
-"Result for spec [" + i + "] of test [" + testItem + "]
doesn't match.",
-// Use Row.equals() to enable equality for complex
structure, i.e. byte[]
-Row.of(testItem.results.get(i)),
-Row.of(row.getField(i)));
+
assertThat(result.getResolvedSchema().getColumnDataTypes().get(i).getLogicalType())
+.as(
+"Logical type for spec ["
++ i
++ "] of test ["
+
[GitHub] [flink] flinkbot edited a comment on pull request #18624: [FLINK-25388]
flinkbot edited a comment on pull request #18624: URL: https://github.com/apache/flink/pull/18624#issuecomment-1029176049 ## CI report: * 608aae7635c45c7c7f391add2a816423623ebd35 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30717) * 079bbdb3081175fa659321c6d8a869c2fec8f885 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18624: [FLINK-25388]
flinkbot edited a comment on pull request #18624: URL: https://github.com/apache/flink/pull/18624#issuecomment-1029176049 ## CI report: * 608aae7635c45c7c7f391add2a816423623ebd35 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30717) * 079bbdb3081175fa659321c6d8a869c2fec8f885 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30743) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #18503: [FLINK-25331][tests] FlinkImageBuilder checks for Java 17
zentol merged pull request #18503: URL: https://github.com/apache/flink/pull/18503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25331) Allow testcontainer tests to run on Java 17
[ https://issues.apache.org/jira/browse/FLINK-25331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-25331. Fix Version/s: 1.15.0 Resolution: Fixed master: 96c2500739bc5d0a0503a165daaf7549a7b6a84c > Allow testcontainer tests to run on Java 17 > --- > > Key: FLINK-25331 > URL: https://issues.apache.org/jira/browse/FLINK-25331 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Tests using testcontainers for Flink are currently locked to Java 8. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25001) Zookeeper 3.4 fails on Java 17
[ https://issues.apache.org/jira/browse/FLINK-25001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-25001. Resolution: Fixed Support for Zookeeper 3.4 has been dropped. > Zookeeper 3.4 fails on Java 17 > -- > > Key: FLINK-25001 > URL: https://issues.apache.org/jira/browse/FLINK-25001 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Chesnay Schepler >Priority: Major > > If we want to support Java 17 then it would be good to drop Zookeeper 3.4 (at > the very least as the default), because it does not support Java 17. > See https://issues.apache.org/jira/browse/ZOOKEEPER-3779 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] pnowojski merged pull request #18575: [FLINK-25827][task] Fix potential memory leak in SourceOperator when using CompletableFuture.anyOf
pnowojski merged pull request #18575: URL: https://github.com/apache/flink/pull/18575 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25958) OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
Victor Xu created FLINK-25958:
-
Summary: OOME Checkpoints & Savepoints were shown as COMPLETE in
Flink UI
Key: FLINK-25958
URL: https://issues.apache.org/jira/browse/FLINK-25958
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.13.5
Environment: Ververica Platform 2.6.2
Flink 1.13.5
Reporter: Victor Xu
Attachments: JIRA-1.jpg
Flink job was running but the checkpoints & savepoints were failing all the
time due to OOM Exception. However, the Flink UI showed COMPLETE for those
checkpoints & savepoints.
For example (checkpoint 39 & 40):
{noformat}
2022-01-27 02:41:39,969 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 39 (type=CHECKPOINT) @ 1643251299952 for job
ab2217e5ce144087bbddf6bd6c3
668eb.
2022-01-27 02:43:19,678 WARN org.apache.flink.runtime.jobmaster.JobMaster
[] - Error while processing AcknowledgeCheckpoint message
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete the
pending checkpoint 39. Failure reason: Failure to finalize checkpoint.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-s
tream2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1054) ~[?:?]
at
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:627)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.serializeCheckpoint(KubernetesHaCheckpointStore.java:204)
~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.09
1138-2.jar:?]
at
com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.addCheckpoint(KubernetesHaCheckpointStore.java:83)
~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.091138-2.
jar:?]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1209)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2022-01-27 03:41:39,970 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 40 (type=CHECKPOINT) @ 1643254899952 for job
ab2217e5ce144087bbddf6bd6c3
668eb.
2022-01-27 03:43:22,326 WARN org.apache.flink.runtime.jobmaster.JobMaster
[] - Error while processing AcknowledgeCheckpoint message
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete the
pending checkpoint 40. Failure reason: Failure to finalize checkpoint.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-s
tream2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
[jira] [Updated] (FLINK-25958) OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
[
https://issues.apache.org/jira/browse/FLINK-25958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-25958:
---
Affects Version/s: 1.14.3
1.12.7
1.15.0
> OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
>
>
> Key: FLINK-25958
> URL: https://issues.apache.org/jira/browse/FLINK-25958
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0, 1.12.7, 1.13.5, 1.14.3
> Environment: Ververica Platform 2.6.2
> Flink 1.13.5
>Reporter: Victor Xu
>Priority: Major
> Attachments: JIRA-1.jpg
>
>
> Flink job was running but the checkpoints & savepoints were failing all the
> time due to OOM Exception. However, the Flink UI showed COMPLETE for those
> checkpoints & savepoints.
> For example (checkpoint 39 & 40):
> {noformat}
> 2022-01-27 02:41:39,969 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 39 (type=CHECKPOINT) @ 1643251299952 for job
> ab2217e5ce144087bbddf6bd6c3
> 668eb.
> 2022-01-27 02:43:19,678 WARN org.apache.flink.runtime.jobmaster.JobMaster
> [] - Error while processing AcknowledgeCheckpoint message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> the pending checkpoint 39. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-s
> tream2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.lang.Thread.run(Thread.java:829) [?:?]
> Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1054) ~[?:?]
> at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:627)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.serializeCheckpoint(KubernetesHaCheckpointStore.java:204)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.09
> 1138-2.jar:?]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.addCheckpoint(KubernetesHaCheckpointStore.java:83)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.091138-2.
> jar:?]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1209)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> ... 9 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2022-01-27 03:41:39,970 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 40 (type=CHECKPOINT) @ 1643254899952 for job
> ab2217e5ce144087bbddf6bd6c3
> 668eb.
> 2022-01-27 03:43:22,326 WARN org.apache.flink.runtime.jobmaster.JobMaster
> [] - Error while processing AcknowledgeCheckpoint message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> the pending checkpoint 40. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
>
[GitHub] [flink] fapaul commented on a change in pull request #18476: [FLINK-25726][streaming] Add Global Committer as post commit topology
fapaul commented on a change in pull request #18476: URL: https://github.com/apache/flink/pull/18476#discussion_r799342227 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitter.java ## @@ -0,0 +1,156 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.api.connector.sink2; + +import org.apache.flink.api.common.state.ListState; +import org.apache.flink.api.common.state.ListStateDescriptor; +import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer; +import org.apache.flink.api.connector.sink2.Committer; +import org.apache.flink.core.io.SimpleVersionedSerializer; +import org.apache.flink.runtime.state.StateInitializationContext; +import org.apache.flink.runtime.state.StateSnapshotContext; +import org.apache.flink.streaming.api.graph.StreamConfig; +import org.apache.flink.streaming.api.operators.AbstractStreamOperator; +import org.apache.flink.streaming.api.operators.BoundedOneInput; +import org.apache.flink.streaming.api.operators.OneInputStreamOperator; +import org.apache.flink.streaming.api.operators.Output; +import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState; +import org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManager; +import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollector; +import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer; +import org.apache.flink.streaming.runtime.operators.sink.committables.CommittableManager; +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord; +import org.apache.flink.streaming.runtime.tasks.StreamTask; +import org.apache.flink.util.function.SerializableSupplier; + +import java.io.IOException; +import java.util.Collection; +import java.util.Collections; + +import static org.apache.flink.util.Preconditions.checkNotNull; + +class GlobalCommitter extends AbstractStreamOperator Review comment: Rename to `GlobalCommitterOperator` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18545: [FLINK-25927][connectors] Make flink-connector-base dependency usage consistent across all connectors
zentol commented on a change in pull request #18545:
URL: https://github.com/apache/flink/pull/18545#discussion_r799342225
##
File path: flink-end-to-end-tests/test-scripts/test_quickstarts.sh
##
@@ -41,7 +41,7 @@ mkdir -p "${TEST_DATA_DIR}"
cd "${TEST_DATA_DIR}"
ARTIFACT_ID=flink-quickstart-${TEST_TYPE}
-ARTIFACT_VERSION=0.1
+ARTIFACT_VERSION=${FLINK_VERSION}
Review comment:
ARTIFACT_VERSION is the version of the generated project. The version of
the archetype is FLINK_VERSION.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dannycranmer commented on a change in pull request #18483: [FLINK-24041][connectors] Removed public setter for elementConverter in As…
dannycranmer commented on a change in pull request #18483:
URL: https://github.com/apache/flink/pull/18483#discussion_r799342056
##
File path:
flink-connectors/flink-connector-aws-kinesis-data-streams/src/test/java/org/apache/flink/connector/kinesis/sink/KinesisDataStreamsSinkBuilderTest.java
##
@@ -60,9 +58,36 @@ public void streamNameOfSinkMustBeSetToNonEmptyWhenBuilt() {
() ->
KinesisDataStreamsSink.builder()
.setStreamName("")
-
.setElementConverter(ELEMENT_CONVERTER_PLACEHOLDER)
+
.setPartitionKeyGenerator(PARTITION_KEY_GENERATOR)
+
.setSerializationSchema(SERIALIZATION_SCHEMA)
.build())
.withMessageContaining(
"The stream name must be set when initializing the KDS
Sink.");
}
+
+@Test
+public void serializationSchemaMustBeSetWhenSinkIsBuilt() {
+Assertions.assertThatExceptionOfType(NullPointerException.class)
+.isThrownBy(
+() ->
+KinesisDataStreamsSink.builder()
+.setStreamName("stream")
+
.setPartitionKeyGenerator(PARTITION_KEY_GENERATOR)
+.build())
+.withMessageContaining(
+"No SerializationSchema was supplied to the
KinesisDataStreamsSinkElementConverter builder.");
+}
+
+@Test
+public void partitionKeyGeneratorMustBeSetWhenSinkIsBuilt() {
+Assertions.assertThatExceptionOfType(NullPointerException.class)
+.isThrownBy(
+() ->
+KinesisDataStreamsSink.builder()
+.setStreamName("stream")
+
.setSerializationSchema(SERIALIZATION_SCHEMA)
+.build())
+.withMessageContaining(
+"No PartitionKeyGenerator lambda was supplied to the
KinesisDataStreamsSinkElementConverter builder.");
Review comment:
nit: It does not have to be a lambda..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #18476: [FLINK-25726][streaming] Add Global Committer as post commit topology
fapaul commented on a change in pull request #18476:
URL: https://github.com/apache/flink/pull/18476#discussion_r799342569
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/StandardSinkTopologies.java
##
@@ -34,7 +41,21 @@ private StandardSinkTopologies() {}
*/
public static void addGlobalCommitter(
DataStream> committables,
-SerializableSupplier> globalCommitterFactory) {
-// TODO: FLINK-25726
+SerializableSupplier> globalCommitterFactory,
Review comment:
Rename to `committerFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25958) OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
[
https://issues.apache.org/jira/browse/FLINK-25958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486941#comment-17486941
]
Piotr Nowojski commented on FLINK-25958:
It looks like the problem is caused by pre-mature reporting that the checkpoint
is completed via {{PendingCheckpointStats#reportCompletedCheckpoint}} call in
{{PendingCheckpoint#finalizeCheckpoint}}. That's when {{PendingCheckpoint}} is
converted to {{CompletedCheckpoint}}, however this doesn't mean the checkpoint
will indeed completed. For example adding to checkpoint store can still fail.
It looks like there is no good reason behind this behaviour and it's just an
unintentional artefact of FLINK-4410 changes, that added support for displaying
failed/pending checkpoints stats.
The most naive solution might be just moving
{{PendingCheckpointStats#reportCompletedCheckpoint}} at the end of the
checkpointing process. However one thing to consider is that this could
indadvertedly create the opposite problem, where checkpoint has completed but
failure while reporting fails to mark it as such.
> OOME Checkpoints & Savepoints were shown as COMPLETE in Flink UI
>
>
> Key: FLINK-25958
> URL: https://issues.apache.org/jira/browse/FLINK-25958
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0, 1.12.7, 1.13.5, 1.14.3
> Environment: Ververica Platform 2.6.2
> Flink 1.13.5
>Reporter: Victor Xu
>Priority: Major
> Attachments: JIRA-1.jpg
>
>
> Flink job was running but the checkpoints & savepoints were failing all the
> time due to OOM Exception. However, the Flink UI showed COMPLETE for those
> checkpoints & savepoints.
> For example (checkpoint 39 & 40):
> {noformat}
> 2022-01-27 02:41:39,969 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 39 (type=CHECKPOINT) @ 1643251299952 for job
> ab2217e5ce144087bbddf6bd6c3
> 668eb.
> 2022-01-27 02:43:19,678 WARN org.apache.flink.runtime.jobmaster.JobMaster
> [] - Error while processing AcknowledgeCheckpoint message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> the pending checkpoint 39. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1227)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-s
> tream2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.lang.Thread.run(Thread.java:829) [?:?]
> Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1054) ~[?:?]
> at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:627)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.serializeCheckpoint(KubernetesHaCheckpointStore.java:204)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.09
> 1138-2.jar:?]
> at
> com.ververica.platform.flink.ha.kubernetes.KubernetesHaCheckpointStore.addCheckpoint(KubernetesHaCheckpointStore.java:83)
> ~[vvp-flink-ha-kubernetes-flink113-1.4-20211013.091138-2.
> jar:?]
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1209)
> ~[flink-dist_2.12-1.13.5-stream2.jar:1.13.5-stream2]
> ... 9 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2022-01-27 03:41:39,970 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 40 (type=CHECKPOINT) @ 1643
[GitHub] [flink] zentol commented on a change in pull request #18545: [FLINK-25927][connectors] Make flink-connector-base dependency usage consistent across all connectors
zentol commented on a change in pull request #18545:
URL: https://github.com/apache/flink/pull/18545#discussion_r799343345
##
File path: flink-end-to-end-tests/test-scripts/test_quickstarts.sh
##
@@ -41,7 +41,7 @@ mkdir -p "${TEST_DATA_DIR}"
cd "${TEST_DATA_DIR}"
ARTIFACT_ID=flink-quickstart-${TEST_TYPE}
-ARTIFACT_VERSION=0.1
+ARTIFACT_VERSION=${FLINK_VERSION}
Review comment:
As such I don't understand why we're changing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18545: [FLINK-25927][connectors] Make flink-connector-base dependency usage consistent across all connectors
zentol commented on a change in pull request #18545: URL: https://github.com/apache/flink/pull/18545#discussion_r799344280 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -86,6 +86,13 @@ under the License. + + org.apache.flink + flink-connector-base Review comment: I don't understand why this dependency is necessary. It should be on the classpath in any case because the connectors have a dependency on it. This should also apply to other modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
flinkbot edited a comment on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1028011200 ## CI report: * 326a658958b6dc91c90ec25f0f501a4947a00c17 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30736) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18625: [FLINK-25947][build] Simplify shade-plugin configuration
flinkbot edited a comment on pull request #18625: URL: https://github.com/apache/flink/pull/18625#issuecomment-1029737971 ## CI report: * 47f8d63ed2f102d08db7e28e0f935340690599e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30733) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18545: [FLINK-25927][connectors] Make flink-connector-base dependency usage consistent across all connectors
zentol commented on a change in pull request #18545:
URL: https://github.com/apache/flink/pull/18545#discussion_r799347081
##
File path: flink-dist/src/main/assemblies/bin.xml
##
@@ -108,6 +108,13 @@ under the License.
+
+
../flink-connectors/flink-connector-base/target/flink-connector-base-${project.version}.jar
Review comment:
This is unnecessary; it could just be included in the flink-dist jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-24970) FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired after 60000 milliseconds while awaiting InitProducerId
[
https://issues.apache.org/jira/browse/FLINK-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24970:
---
Labels: stale-assigned test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired after
> 6 milliseconds while awaiting InitProducerId
> ---
>
> Key: FLINK-24970
> URL: https://issues.apache.org/jira/browse/FLINK-24970
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Assignee: Fabian Paul
>Priority: Critical
> Labels: stale-assigned, test-stability
>
> {code:java}
> Nov 20 23:27:52 [ERROR] Tests run: 15, Failures: 10, Errors: 1, Skipped: 0,
> Time elapsed: 341.449 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> Nov 20 23:27:52 [ERROR] testScaleUpAfterScalingDown Time elapsed: 75.789 s
> <<< ERROR!
> Nov 20 23:27:52 org.apache.kafka.common.errors.TimeoutException: Timeout
> expired after 6 milliseconds while awaiting InitProducerId{code}
> After that a lot of tests failed due to
> {code:java}
> Nov 20 23:27:52 [ERROR] testScaleUpAfterScalingDown Time elapsed: 75.789 s
> <<< ERROR!
> Nov 20 23:27:52 org.apache.kafka.common.errors.TimeoutException: Timeout
> expired after 6 milliseconds while awaiting InitProducerId
> Nov 20 23:27:52
> Nov 20 23:27:52 [ERROR] testFailAndRecoverSameCheckpointTwice Time elapsed:
> 10.031 s <<< FAILURE!
> Nov 20 23:27:52 java.lang.AssertionError: Detected producer leak. Thread
> name: kafka-producer-network-thread |
> producer-MockTask-002a002c-9
> Nov 20 23:27:52 at org.junit.Assert.fail(Assert.java:89) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24674) Create corresponding resouces for task manager Pods
[ https://issues.apache.org/jira/browse/FLINK-24674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-24674: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Create corresponding resouces for task manager Pods > --- > > Key: FLINK-24674 > URL: https://issues.apache.org/jira/browse/FLINK-24674 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available, stale-assigned > > When creating Flink session cluster on K8S, Flink job manager won't create > configmap for the Hadoop configuration stored in the Docker image. > Then when users submit Java application, the task manager Pods will not be > abled to created, because it fails to mount the expected configmap of Hadoop > configuration. > This patch fixes this issue. So Flink users can include Hadoop classes and > configuration in Docker image. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24661) ConfigOption add isSecret method to judge sensitive options
[ https://issues.apache.org/jira/browse/FLINK-24661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-24661: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > ConfigOption add isSecret method to judge sensitive options > --- > > Key: FLINK-24661 > URL: https://issues.apache.org/jira/browse/FLINK-24661 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 1.13.3 >Reporter: Ada Wong >Assignee: Ada Wong >Priority: Major > Labels: pull-request-available, stale-assigned > > Related ticket https://issues.apache.org/jira/browse/FLINK-24381 > [~chesnay] > Should this ticket modify connectors for example ES and JDBC password option, > or just only modify core module not connector modules? > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22551) checkpoints: strange behaviour
[
https://issues.apache.org/jira/browse/FLINK-22551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22551:
---
Labels: auto-deprioritized-critical auto-deprioritized-major
auto-deprioritized-minor (was: auto-deprioritized-critical
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> checkpoints: strange behaviour
> ---
>
> Key: FLINK-22551
> URL: https://issues.apache.org/jira/browse/FLINK-22551
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
> Environment: {code:java}
> java -version
> openjdk version "11.0.2" 2019-01-15
> OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
> {code}
>Reporter: buom
>Priority: Not a Priority
> Labels: auto-deprioritized-critical, auto-deprioritized-major,
> auto-deprioritized-minor
>
> * +*Case 1*:+ Work as expected
> {code:java}
> public class Example {
> public static class ExampleSource extends RichSourceFunction
> implements CheckpointedFunction {
> private volatile boolean isRunning = true;
> @Override
> public void open(Configuration parameters) throws Exception {
> System.out.println("[source] invoke open()");
> }
> @Override
> public void close() throws Exception {
> isRunning = false;
> System.out.println("[source] invoke close()");
> }
> @Override
> public void run(SourceContext ctx) throws Exception {
> System.out.println("[source] invoke run()");
> while (isRunning) {
> ctx.collect("Flink");
> Thread.sleep(500);
> }
> }
> @Override
> public void cancel() {
> isRunning = false;
> System.out.println("[source] invoke cancel()");
> }
> @Override
> public void snapshotState(FunctionSnapshotContext context) throws
> Exception {
> System.out.println("[source] invoke snapshotState()");
> }
> @Override
> public void initializeState(FunctionInitializationContext context)
> throws Exception {
> System.out.println("[source] invoke initializeState()");
> }
> }
> public static class ExampleSink extends PrintSinkFunction
> implements CheckpointedFunction {
> @Override
> public void snapshotState(FunctionSnapshotContext context) throws
> Exception {
> System.out.println("[sink] invoke snapshotState()");
> }
> @Override
> public void initializeState(FunctionInitializationContext context)
> throws Exception {
> System.out.println("[sink] invoke initializeState()");
> }
> }
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env =
>
> StreamExecutionEnvironment.getExecutionEnvironment().enableCheckpointing(1000);
> DataStream stream = env.addSource(new ExampleSource());
> stream.addSink(new ExampleSink()).setParallelism(1);
> env.execute();
> }
> }
> {code}
> {code:java}
> $ java -jar ./example.jar
> [sink] invoke initializeState()
> [source] invoke initializeState()
> [source] invoke open()
> [source] invoke run()
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> Flink
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> Flink
> Flink
> [sink] invoke snapshotState()
> [source] invoke snapshotState()
> ^C
> {code}
> * *+Case 2:+* Run as unexpected (w/ _parallelism = 1_)
> {code:java}
> public class Example {
> public static class ExampleSource extends RichSourceFunction
> implements CheckpointedFunction {
> private volatile boolean isRunning = true;
> @Override
> public void open(Configuration parameters) throws Exception {
> System.out.println("[source] invoke open()");
> }
> @Override
> public void close() throws Exception {
> isRunning = false;
> System.out.println("[source] invoke close()");
> }
> @Override
> public void run(SourceContext ctx) throws Exception {
> System.out.println("[source] invoke run()");
> while (isRunning) {
> ctx.collect("Flink");
> Thread.sleep(500);
> }
> }
> @Override
> public void cancel() {
>
