[GitHub] [flink] flinkbot commented on pull request #18498: [FLINK-25801][ metrics]add cpu processor metric of taskmanager
flinkbot commented on pull request #18498: URL: https://github.com/apache/flink/pull/18498#issuecomment-1020905745 ## CI report: * e6d959da396523b05fdc25889fd2f1859857c70e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on pull request #18454: [hotfix][connector/pulsar] Fix typo in JavaDocs example.
deadwind4 commented on pull request #18454: URL: https://github.com/apache/flink/pull/18454#issuecomment-1020906388 @syhily Could you review this? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25154) FLIP-193: Snapshots ownership
[ https://issues.apache.org/jira/browse/FLINK-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481627#comment-17481627 ] Dawid Wysakowicz commented on FLINK-25154: -- Sorry, [~monster#12] I missed the notification. This is/was high on our roadmap and we prefer to keep it "in-house", hope that's fine with you. > FLIP-193: Snapshots ownership > - > > Key: FLINK-25154 > URL: https://issues.apache.org/jira/browse/FLINK-25154 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.15.0 > > > Task for implementing FLIP-193: https://cwiki.apache.org/confluence/x/bIyqCw -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25803) Implement partition and bucket filter of FileStoreScanImpl
Caizhi Weng created FLINK-25803: --- Summary: Implement partition and bucket filter of FileStoreScanImpl Key: FLINK-25803 URL: https://issues.apache.org/jira/browse/FLINK-25803 Project: Flink Issue Type: Sub-task Components: Table Store Reporter: Caizhi Weng Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18498: [FLINK-25801][ metrics]add cpu processor metric of taskmanager
flinkbot edited a comment on pull request #18498: URL: https://github.com/apache/flink/pull/18498#issuecomment-1020905745 ## CI report: * e6d959da396523b05fdc25889fd2f1859857c70e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30123) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper opened a new pull request #14: [FLINK-25803] Implement partition and bucket filter in FileStoreScanImpl
tsreaper opened a new pull request #14: URL: https://github.com/apache/flink-table-store/pull/14 This PR implements partition and bucket filter in `FileStoreScanImpl`. It also perform some small optimizations such as concurrent reads in `FileStoreScanImpl`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25803) Implement partition and bucket filter of FileStoreScanImpl
[ https://issues.apache.org/jira/browse/FLINK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25803: --- Labels: pull-request-available (was: ) > Implement partition and bucket filter of FileStoreScanImpl > -- > > Key: FLINK-25803 > URL: https://issues.apache.org/jira/browse/FLINK-25803 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25803) Implement partition and bucket filter of FileStoreScanImpl
[ https://issues.apache.org/jira/browse/FLINK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-25803: Assignee: Caizhi Weng > Implement partition and bucket filter of FileStoreScanImpl > -- > > Key: FLINK-25803 > URL: https://issues.apache.org/jira/browse/FLINK-25803 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) * 56fee29c28007e3ba03d1f14ed7ef12a38a12b0d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25803) Implement partition and bucket filter of FileStoreScanImpl
[ https://issues.apache.org/jira/browse/FLINK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng updated FLINK-25803: Description: Implements partition and bucket filter in FileStoreScanImpl. Also perform some small optimizations such as concurrent reads in FileStoreScanImpl. > Implement partition and bucket filter of FileStoreScanImpl > -- > > Key: FLINK-25803 > URL: https://issues.apache.org/jira/browse/FLINK-25803 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Implements partition and bucket filter in FileStoreScanImpl. > Also perform some small optimizations such as concurrent reads in > FileStoreScanImpl. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25803) Implement partition and bucket filter of FileStoreScanImpl
[ https://issues.apache.org/jira/browse/FLINK-25803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng updated FLINK-25803: Description: Implement partition and bucket filter in FileStoreScanImpl. Also perform some small optimizations such as concurrent reads in FileStoreScanImpl. was: Implements partition and bucket filter in FileStoreScanImpl. Also perform some small optimizations such as concurrent reads in FileStoreScanImpl. > Implement partition and bucket filter of FileStoreScanImpl > -- > > Key: FLINK-25803 > URL: https://issues.apache.org/jira/browse/FLINK-25803 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Implement partition and bucket filter in FileStoreScanImpl. > Also perform some small optimizations such as concurrent reads in > FileStoreScanImpl. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18460: [FLINK-25767][doc] Totally translated state.md into Chinese
flinkbot edited a comment on pull request #18460: URL: https://github.com/apache/flink/pull/18460#issuecomment-1019844798 ## CI report: * 0a08d1b660690c6c98f73066e1a54bfff10c20be Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30094) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] renxiaoyao opened a new pull request #18499: [hotfix][docs] Fixed error for Hive ZH docs.
renxiaoyao opened a new pull request #18499: URL: https://github.com/apache/flink/pull/18499 ## What is the purpose of the change *Fixed error for Hive ZH docs.* ## Brief change log - *"一下"* updated to *"以下"* ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791455950
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
Review comment:
> Snapshot creation time: (isn't that just preparing for changelogs to
be uploaded)?
> And previously snapshot creat time: flush + prepare SSTs to be uploaded?
Yes.
> Would these two parts significantly different?
> I think most of the time reduced is from the async phase
For changelog, it's usually under 100ms, without it can take tens of seconds
(above p99%). That can sound not much, but this phase blocks the entire
processing. I describe it
[here](https://github.com/apache/flink/pull/18431#discussion_r790436952).
But actually this statement is about reduction by using Async checkpoints
which were implemented long time ago, not related to the Changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791456268
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
+
+With Changelog enabled, Flink uploads state changes continuously, forming a
changelog. On checkpoint, only the relevant
+part of this changelog needs to be uploaded. Independently, configured state
backend is checkpointed in the
+background periodically. Upon successful upload, changelog is truncated.
+
+As a result, asynchronous phase is reduced, as well as synchronous phase (in
particular, long-tail).
Review comment:
Explained, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18499: [hotfix][docs] Fixed error for Hive ZH docs.
flinkbot commented on pull request #18499: URL: https://github.com/apache/flink/pull/18499#issuecomment-1020917616 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 186a3b88de92d0efd17497564d5755f89822102d (Tue Jan 25 08:17:32 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791458494
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
Review comment:
I've expanded this section to address (2), do you still have any
concerns about (1)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18499: [hotfix][docs] Fixed error for Hive ZH docs.
flinkbot commented on pull request #18499: URL: https://github.com/apache/flink/pull/18499#issuecomment-1020919615 ## CI report: * 186a3b88de92d0efd17497564d5755f89822102d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) * 56fee29c28007e3ba03d1f14ed7ef12a38a12b0d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18499: [hotfix][docs] Fixed error for Hive ZH docs.
flinkbot edited a comment on pull request #18499: URL: https://github.com/apache/flink/pull/18499#issuecomment-1020919615 ## CI report: * 186a3b88de92d0efd17497564d5755f89822102d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30124) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr closed pull request #18290: [FLINK-25391][connectors][formats] Update existing table factories/format factories for catalog table options forwarding
twalthr closed pull request #18290: URL: https://github.com/apache/flink/pull/18290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25391) Updating existing table factories for mutable table options
[ https://issues.apache.org/jira/browse/FLINK-25391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-25391. Fix Version/s: 1.15.0 Resolution: Fixed Fixed in master: commit fa161d3c5636370f5129320a9ca464e38f88fc6f [format-json] Forward catalog table options commit 0370c36eff86a0af9485405ca3a51663c33cbadf [format-csv] Forward catalog table options commit 8f77862fa5ecbec5ee26b7b2b68478ad50943a3e [format-avro] Forward catalog table options commit 5175ed0d48835344c1cd4282372d6b01571d914b [connector-hbase] Forward catalog table options commit 2cb86ff03747499bfceda74cb8cc1ea48c385452 [connector-kinesis] Forward catalog table options commit 0c34c994f9906e58963f85739fc951221b11d26a [connector-kafka] Forward catalog table options commit c9260311637ad47a6e67f154c629ddd49d9f262a [connector-files] Forward catalog table options commit c61162b30f4b5567ecc2ee29481fcc87e5016428 [connector-jdbc] Forward catalog table options commit 5e28f66f6ef2ed03f9ee69148fe5079ae5e358c4 [connector-elasticsearch] Forward catalog table options > Updating existing table factories for mutable table options > --- > > Key: FLINK-25391 > URL: https://issues.apache.org/jira/browse/FLINK-25391 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Timo Walther >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Update all existing factories for FLINK-25390. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dianfu commented on a change in pull request #18418: [FLINK-25719][python] Support General Python UDF in Thread Mode
dianfu commented on a change in pull request #18418:
URL: https://github.com/apache/flink/pull/18418#discussion_r790681717
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/utils/CommonPythonUtil.java
##
@@ -158,6 +158,20 @@ public static boolean
isPythonWorkerUsingManagedMemory(Configuration config) {
}
}
+@SuppressWarnings("unchecked")
+public static boolean isPythonWorkerInProcessMode(Configuration config) {
+Class clazz = loadClass("org.apache.flink.python.PythonOptions");
+try {
+return config.getString(
+(ConfigOption)
+
(clazz.getField("PYTHON_EXECUTION_MODE").get(null)))
+.equals("process");
Review comment:
```suggestion
.equalsIgnoreCase("process");
```
##
File path:
flink-python/src/main/java/org/apache/flink/python/env/pemja/EmbeddedPythonEnvironmentManager.java
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.python.env.pemja;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.python.env.AbstractPythonEnvironmentManager;
+import org.apache.flink.python.env.PythonDependencyInfo;
+import org.apache.flink.python.env.PythonEnvironment;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import pemja.core.PythonInterpreterConfig;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The base class of python environment manager which is used to create the
PythonEnvironment
+ * object. It's used to run python UDF in embedded Python environment.
+ */
+@Internal
+public class EmbeddedPythonEnvironmentManager extends
AbstractPythonEnvironmentManager {
+
+private static final Logger LOG =
+LoggerFactory.getLogger(EmbeddedPythonEnvironmentManager.class);
+
+public EmbeddedPythonEnvironmentManager(
+PythonDependencyInfo dependencyInfo,
+String[] tmpDirectories,
+Map systemEnv,
+JobID jobID) {
+super(dependencyInfo, tmpDirectories, systemEnv, jobID);
+}
+
+@Override
+public PythonEnvironment createEnvironment() throws Exception {
+Map env = new HashMap<>(resource.env);
+
+PythonInterpreterConfig.ExecType execType;
+if (dependencyInfo.getThreadModeExecType().equals("sub-interpreter")) {
Review comment:
```suggestion
if
(dependencyInfo.getThreadModeExecType().equalsIgnoreCase("sub-interpreter")) {
```
##
File path:
flink-python/src/main/java/org/apache/flink/streaming/api/operators/python/AbstractEmbeddedPythonFunctionOperator.java
##
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators.python;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.fnexecution.v1.FlinkFnApi;
+import org.apache.flink.python.PythonConfig;
+import org.apache.flink.python.env.PythonDependencyInfo;
+import org.apache.flink.python.env.pemja.EmbeddedPythonEnvironment;
+import org.apache.flink.python.env.pemja.EmbeddedPythonEn
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791467158
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
+
+With Changelog enabled, Flink uploads state changes continuously, forming a
changelog. On checkpoint, only the relevant
+part of this changelog needs to be uploaded. Independently, configured state
backend is checkpointed in the
+background periodically. Upon successful upload, changelog is truncated.
+
+As a result, asynchronous phase is reduced, as well as synchronous phase (in
particular, long-tail).
+
+On the flip side, resource usage is higher:
+
+- more files are created on DFS
+- more IO bandwidth is used to upload
+- more CPU used to serialize state changes
+- more memory used by Task Managers to buffer state changes
+- todo: more details after testing, maybe link to blogpost
+
+Recovery time is another thing to consider. Depending on the
`state.backend.changelog.periodic-materialize.interval`,
+changelog can become lengthy and replaying it may take more time. However,
recovery time combined with checkpoint
+duration will likely be still lower than in non-changelog setup, providing
lower end-to-end latency even in failover
+case.
Review comment:
> I also saw a lot of cases that recovery time increased by tens of
seconds but checkpoint duration does not decrease that much.
If checkpoint duration doesn't decrease significantly then it's probably not
a suitable case to use Changelog. So the question I think is what's the ratio
between these times and how to put it in the docs.
> I do not think we should mix recovery time with checkpoint duration here.
Reducing **effective** recovery time is actually one of the goals of the
project (please see FLIP
[Motivation](https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints#FLIP158:Generalizedincrementalcheckpoints-Motivation)
and discussions); and the mean to achieve this is by having less data to
replay for the whole pipeline (while having data to replay by Changelog on Task
level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #18431: [FLINK-25024][docs] Add Changelog backend docs
rkhachatryan commented on a change in pull request #18431:
URL: https://github.com/apache/flink/pull/18431#discussion_r791456268
##
File path: docs/content/docs/ops/state/state_backends.md
##
@@ -325,6 +325,126 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
{{< top >}}
+## Enabling Changelog
+
+// todo: Chinese version of all changed docs
+
+// todo: mention in [large state tuning]({{< ref
"docs/ops/state/large_state_tuning" >}})? or 1.16?
+
+{{< hint warning >}} The feature is in experimental status. {{< /hint >}}
+
+{{< hint warning >}} Enabling Changelog may have a negative performance impact
on your application (see below). {{< /hint >}}
+
+### Introduction
+
+Changelog is a feature that aims to decrease checkpointing time, and therefore
end-to-end latency in exactly-once mode.
+
+Most commonly, checkpoint duration is affected by:
+
+1. Barrier travel time and alignment, addressed by
+ [Unaligned checkpoints]({{< ref
"docs/ops/state/checkpointing_under_backpressure#unaligned-checkpoints" >}})
+ and [Buffer debloating]({{< ref
"docs/ops/state/checkpointing_under_backpressure#buffer-debloating" >}})
+2. Snapshot creation time (so-called synchronous phase), addressed by
Asynchronous snapshots
+3. Snapshot upload time (asynchronous phase)
+
+The latter (upload time) can be decreased by [Incremental checkpoints]({{< ref
"#incremental-checkpoints" >}}). However,
+even with Incremental checkpoints, large deployments tend to have at least one
task in every checkpoint that uploads a
+lot of data (e.g. after compaction).
+
+With Changelog enabled, Flink uploads state changes continuously, forming a
changelog. On checkpoint, only the relevant
+part of this changelog needs to be uploaded. Independently, configured state
backend is checkpointed in the
+background periodically. Upon successful upload, changelog is truncated.
+
+As a result, asynchronous phase is reduced, as well as synchronous phase (in
particular, long-tail).
Review comment:
Sure, explained, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zjureel commented on pull request #18490: [FLINK-25794][sql-runtime] Clean cache after memory segments in it after they are released to MemoryManager
zjureel commented on pull request #18490: URL: https://github.com/apache/flink/pull/18490#issuecomment-1020930174 Thanks @JingsongLi , I think I miss the operations in `MemoryManager` :( And another issue is in `BinaryHashTableTest.testBinaryHashBucketAreaNotEnoughMem`, there's no `table.free` after `table.close`, I think it should be and it causes memory leaks in my cache memory manager. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25753) KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao reassigned FLINK-25753:
---
Assignee: Fabian Paul
> KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP
> -
>
> Key: FLINK-25753
> URL: https://issues.apache.org/jira/browse/FLINK-25753
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Fabian Paul
>Priority: Critical
> Labels: test-stability
>
> The test {{KafkaTableITCase.testStartFromGroupOffsetsLatest}} fails on AZP
> with
> {code}
> 2022-01-21T08:48:26.7044016Z Jan 21 08:48:26 [ERROR]
> KafkaTableITCase.testStartFromGroupOffsetsLatest Time elapsed: 5.308 s <<<
> ERROR!
> 2022-01-21T08:48:26.7044630Z Jan 21 08:48:26
> java.util.concurrent.TimeoutException: Can not get the expected result.
> 2022-01-21T08:48:26.7047268Z Jan 21 08:48:26 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:214)
> 2022-01-21T08:48:26.7048108Z Jan 21 08:48:26 at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:230)
> 2022-01-21T08:48:26.7048836Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.waitingExpectedResults(KafkaTableTestUtils.java:82)
> 2022-01-21T08:48:26.7049632Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsets(KafkaTableITCase.java:942)
> 2022-01-21T08:48:26.7050427Z Jan 21 08:48:26 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsetsLatest(KafkaTableITCase.java:831)
> 2022-01-21T08:48:26.7051077Z Jan 21 08:48:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-01-21T08:48:26.7051638Z Jan 21 08:48:26 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-01-21T08:48:26.7052284Z Jan 21 08:48:26 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-01-21T08:48:26.7053008Z Jan 21 08:48:26 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-01-21T08:48:26.7053681Z Jan 21 08:48:26 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-01-21T08:48:26.7054328Z Jan 21 08:48:26 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-01-21T08:48:26.7054976Z Jan 21 08:48:26 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-01-21T08:48:26.7055621Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-01-21T08:48:26.7056251Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-01-21T08:48:26.7056904Z Jan 21 08:48:26 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-01-21T08:48:26.7057520Z Jan 21 08:48:26 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-01-21T08:48:26.7058445Z Jan 21 08:48:26 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-01-21T08:48:26.7059155Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-01-21T08:48:26.7059779Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-01-21T08:48:26.7060382Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-01-21T08:48:26.7061036Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-01-21T08:48:26.7061670Z Jan 21 08:48:26 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-01-21T08:48:26.7062369Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-01-21T08:48:26.7063456Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-01-21T08:48:26.7064218Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-01-21T08:48:26.7064805Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-01-21T08:48:26.7065375Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-01-21T08:48:26.7065919Z Jan 21 08:48:26 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-01-21T08:48:26.7066437Z Jan 21 08:48:26 at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2022-01-21T08:48:26.7066934Z Jan 21 08:48:26 at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 20
[jira] [Assigned] (FLINK-25673) AdaptiveSchedulerITCase.testStopWithSavepointFailOnFirstSavepointSucceedOnSecond hangs on AZP
[
https://issues.apache.org/jira/browse/FLINK-25673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao reassigned FLINK-25673:
---
Assignee: David Morávek
> AdaptiveSchedulerITCase.testStopWithSavepointFailOnFirstSavepointSucceedOnSecond
> hangs on AZP
> -
>
> Key: FLINK-25673
> URL: https://issues.apache.org/jira/browse/FLINK-25673
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: David Morávek
>Priority: Critical
> Labels: test-stability
>
> The test
> {{AdaptiveSchedulerITCase.testStopWithSavepointFailOnFirstSavepointSucceedOnSecond}}
> hangs on AZP:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29512&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=18044
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) * 56fee29c28007e3ba03d1f14ed7ef12a38a12b0d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #18418: [FLINK-25719][python] Support General Python UDF in Thread Mode
HuangXingBo commented on a change in pull request #18418:
URL: https://github.com/apache/flink/pull/18418#discussion_r791473431
##
File path: flink-python/src/main/java/org/apache/flink/python/PythonOptions.java
##
@@ -214,4 +214,25 @@
+ "in each batch when iterating a Python
MapState. Note that this is an experimental flag "
+ "and might not be available "
+ "in future releases.");
+
+/** Specify the python runtime execution mode. */
+@Experimental
+public static final ConfigOption PYTHON_EXECUTION_MODE =
+ConfigOptions.key("python.execution-mode")
+.defaultValue("process")
Review comment:
Make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25795) Support Pulsar sink connector in Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-25795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481648#comment-17481648 ] Martijn Visser commented on FLINK-25795: [~ana4] We need to take the recent decision to [externalize connectors|https://cwiki.apache.org/confluence/display/FLINK/Connectors] into consideration. Before adding more tech debt, we need to establish a way so that connectors can be externalized and used in Python. > Support Pulsar sink connector in Python DataStream API. > --- > > Key: FLINK-25795 > URL: https://issues.apache.org/jira/browse/FLINK-25795 > Project: Flink > Issue Type: New Feature > Components: API / Python, Connectors / Pulsar >Affects Versions: 1.14.3 >Reporter: Ada Wong >Priority: Major > > https://issues.apache.org/jira/browse/FLINK-20732 the PR of this ticket is > reviewd, we could develop Python Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * de880af98f24b8a8195f65bca492883ce4c05846 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30015) * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) * 56fee29c28007e3ba03d1f14ed7ef12a38a12b0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30126) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a change in pull request #14: [FLINK-25803] Implement partition and bucket filter in FileStoreScanImpl
JingsongLi commented on a change in pull request #14:
URL: https://github.com/apache/flink-table-store/pull/14#discussion_r791468220
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/FileStoreScanImpl.java
##
@@ -18,62 +18,129 @@
package org.apache.flink.table.store.file.operation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.store.file.FileFormat;
import org.apache.flink.table.store.file.Snapshot;
import org.apache.flink.table.store.file.manifest.ManifestEntry;
import org.apache.flink.table.store.file.manifest.ManifestFile;
import org.apache.flink.table.store.file.manifest.ManifestFileMeta;
import org.apache.flink.table.store.file.manifest.ManifestList;
+import org.apache.flink.table.store.file.predicate.And;
+import org.apache.flink.table.store.file.predicate.Equal;
+import org.apache.flink.table.store.file.predicate.Literal;
+import org.apache.flink.table.store.file.predicate.Or;
import org.apache.flink.table.store.file.predicate.Predicate;
import org.apache.flink.table.store.file.utils.FileStorePathFactory;
+import org.apache.flink.table.types.logical.RowType;
import org.apache.flink.util.Preconditions;
import javax.annotation.Nullable;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
/** Default implementation of {@link FileStoreScan}. */
public class FileStoreScanImpl implements FileStoreScan {
+private final RowType partitionType;
+private final RowType keyType;
+private final RowType rowType;
+private final FileFormat fileFormat;
private final FileStorePathFactory pathFactory;
-private final ManifestFile manifestFile;
+
+private final List partitionFieldGetters;
private final ManifestList manifestList;
private Long snapshotId;
private List manifests;
+private Predicate partitionFilter;
+private Predicate keyFilter;
+private Predicate valueFilter;
+private Integer bucket;
public FileStoreScanImpl(
-FileStorePathFactory pathFactory,
-ManifestFile manifestFile,
-ManifestList manifestList) {
+RowType partitionType,
+RowType keyType,
+RowType rowType,
+FileFormat fileFormat,
+FileStorePathFactory pathFactory) {
+this.partitionType = partitionType;
+this.keyType = keyType;
+this.rowType = rowType;
+this.fileFormat = fileFormat;
this.pathFactory = pathFactory;
-this.manifestFile = manifestFile;
-this.manifestList = manifestList;
+
+this.partitionFieldGetters =
Review comment:
Can we extract `RowToArrayConverter`?
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/FileStoreScan.java
##
@@ -31,6 +32,8 @@
FileStoreScan withPartitionFilter(Predicate predicate);
+FileStoreScan withPartitionFilter(List partitions);
Review comment:
`inPartitions`?
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/utils/FileStorePathFactory.java
##
@@ -106,10 +115,12 @@ public SstPathFactory createSstPathFactory(BinaryRowData
partition, int bucket)
}
public String getPartitionString(BinaryRowData partition) {
-return PartitionPathUtils.generatePartitionPath(
-partitionComputer.generatePartValues(
-Preconditions.checkNotNull(
-partition, "Partition row data is null. This
is unexpected.")));
+synchronized (partitionComputer) {
Review comment:
Why needs add `synchronized`?
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/FileStoreScanImpl.java
##
@@ -109,34 +176,72 @@ public Long snapshotId() {
}
private List scan() {
-Map map = new
LinkedHashMap<>();
-for (ManifestFileMeta manifest : manifests) {
-// TODO read each manifest file concurrently
-for (ManifestEntry entry : manifestFile.read(manifest.fileName()))
{
-ManifestEntry.Identifier identifier = entry.identifier();
-switch (entry.kind()) {
-case ADD:
-Preconditions.checkState(
-!map.containsKey(identifier),
-"Trying to add file %s which is already added.
"
-+ "Manifest might be corrupted.",
-
[jira] [Updated] (FLINK-25802) OverWindow in batch mode failed
[
https://issues.apache.org/jira/browse/FLINK-25802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-25802:
---
Component/s: Table SQL / Planner
Table SQL / Runtime
> OverWindow in batch mode failed
> ---
>
> Key: FLINK-25802
> URL: https://issues.apache.org/jira/browse/FLINK-25802
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
>Affects Versions: 1.14.0
>Reporter: Zoyo Pei
>Priority: Major
>
> {code:java}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
> DataStream userStream = env
> .fromElements(
> Row.of(LocalDateTime.parse("2021-08-21T13:00:00"), 1,
> "Alice"),
> Row.of(LocalDateTime.parse("2021-08-21T13:05:00"), 2, "Bob"),
> Row.of(LocalDateTime.parse("2021-08-21T13:10:00"), 2, "Bob"))
> .returns(
> Types.ROW_NAMED(
> new String[]{"ts", "uid", "name"},
> Types.LOCAL_DATE_TIME, Types.INT, Types.STRING));
> tEnv.createTemporaryView(
> "UserTable",
> userStream,
> Schema.newBuilder()
> .column("ts", DataTypes.TIMESTAMP(3))
> .column("uid", DataTypes.INT())
> .column("name", DataTypes.STRING())
> .watermark("ts", "ts - INTERVAL '1' SECOND")
> .build());
> String statement = "SELECT name, ts, COUNT(name) OVER w AS cnt FROM UserTable
> " +
> "WINDOW w AS (" +
> " PARTITION BY name" +
> " ORDER BY ts" +
> " RANGE BETWEEN INTERVAL '10' MINUTE PRECEDING AND CURRENT ROW" +
> ")";
> tEnv.executeSql(statement).print();
> {code}
>
> {code:java}
> /* 1 */
> /* 2 */ public class RangeBoundComparator$38 implements
> org.apache.flink.table.runtime.generated.RecordComparator {
> /* 3 */
> /* 4 */ private final Object[] references;
> /* 5 */
> /* 6 */
> /* 7 */ public RangeBoundComparator$38(Object[] references) {
> /* 8 */ this.references = references;
> /* 9 */
> /* 10 */
> /* 11 */ }
> /* 12 */
> /* 13 */ @Override
> /* 14 */ public int compare(org.apache.flink.table.data.RowData in1,
> org.apache.flink.table.data.RowData in2) {
> /* 15 */
> /* 16 */ org.apache.flink.table.data.TimestampData field$39;
> /* 17 */ boolean isNull$39;
> /* 18 */ org.apache.flink.table.data.TimestampData field$40;
> /* 19 */ boolean isNull$40;
> /* 20 */ isNull$39 = in1.isNullAt(0);
> /* 21 */ field$39 = null;
> /* 22 */ if (!isNull$39) {
> /* 23 */ field$39 = in1.getTimestamp(0, 3);
> /* 24 */ }
> /* 25 */ isNull$40 = in2.isNullAt(0);
> /* 26 */ field$40 = null;
> /* 27 */ if (!isNull$40) {
> /* 28 */ field$40 = in2.getTimestamp(0, 3);
> /* 29 */ }
> /* 30 */ if (isNull$39 && isNull$40) {
> /* 31 */ return 1;
> /* 32 */ } else if (isNull$39 || isNull$40) {
> /* 33 */ return -1;
> /* 34 */ } else {
> /* 35 */
> /* 36 */
> /* 37 */ long result$41;
> /* 38 */ boolean isNull$41;
> /* 39 */ long result$42;
> /* 40 */ boolean isNull$42;
> /* 41 */ boolean isNull$43;
> /* 42 */ long result$44;
> /* 43 */ boolean isNull$45;
> /* 44 */ boolean result$46;
> /* 45 */ isNull$41 = (java.lang.Long) field$39 ==
> null;
> /* 46 */ result$41 = -1L;
> /* 47 */ if (!isNull$41) {
> /* 48 */ result$41 = (java.lang.Long) field$39;
> /* 49 */ }
> /* 50 */ isNull$42 = (java.lang.Long) field$40 ==
> null;
> /* 51 */ result$42 = -1L;
> /* 52 */ if (!isNull$42) {
> /* 53 */ result$42 = (java.lang.Long) field$40;
> /* 54 */ }
> /* 55 */
> /* 56 */
> /* 57 */
> /* 58 */
> /* 59 */ isNull
[GitHub] [flink] XComp commented on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
XComp commented on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1020945550 Thanks for the update @KarmaGYZ. I brought it up in today's release call mentioning that FLINK-25433 (which is a subtask of the efforts around introducing the `JobResultStore`; [FLIP-194](https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore)) is blocked on this PR. But that said, I also underlined that FLINK-25433 is more like a nice-to-have feature as it's not blocking the integration of [FLIP-194](https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore) in general. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
KarmaGYZ commented on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1020946947 @zjureel I think we need also ignore the `maximumCacheSizeBytes` in this case. However, there might be a lot of "if...else" in `FileExecutionGraphInfoStore`. Maybe we should return to a separate `MemoryExecutionGraphInfoStore`. Sorry for the inconvenience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16222) Use plugins mechanism for initializing MetricReporters
[ https://issues.apache.org/jira/browse/FLINK-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481658#comment-17481658 ] Ada Wong commented on FLINK-16222: -- [~ana4] > Use plugins mechanism for initializing MetricReporters > -- > > Key: FLINK-16222 > URL: https://issues.apache.org/jira/browse/FLINK-16222 > Project: Flink > Issue Type: New Feature > Components: Runtime / Metrics >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/FLINK-11952 introduced Plugins > mechanism into Flink. Metrics reporters initialization mechanism can profit > from using this new functionality. Instead of placing MetricsReporters JARs > into /libs, it should be additionally possible (and encouraged) to convert > them into plugins and use the /plugins folder for initialization via > independent plugin classloaders. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (FLINK-16222) Use plugins mechanism for initializing MetricReporters
[ https://issues.apache.org/jira/browse/FLINK-16222 ] Ada Wong deleted comment on FLINK-16222: -- was (Author: ana4): [~ana4] > Use plugins mechanism for initializing MetricReporters > -- > > Key: FLINK-16222 > URL: https://issues.apache.org/jira/browse/FLINK-16222 > Project: Flink > Issue Type: New Feature > Components: Runtime / Metrics >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/FLINK-11952 introduced Plugins > mechanism into Flink. Metrics reporters initialization mechanism can profit > from using this new functionality. Instead of placing MetricsReporters JARs > into /libs, it should be additionally possible (and encouraged) to convert > them into plugins and use the /plugins folder for initialization via > independent plugin classloaders. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] twalthr closed pull request #18485: [FLINK-25791][table-planner] Compact ObjectIdentifier json representation
twalthr closed pull request #18485: URL: https://github.com/apache/flink/pull/18485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingGe commented on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
JingGe commented on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1020958184 Many thanks for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingGe commented on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
JingGe commented on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1020958323 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25791) Make ObjectIdentifier json representation simpler
[
https://issues.apache.org/jira/browse/FLINK-25791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-25791.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master:
commit 04f8724d088e791f0515f4ee06a8c82f4caece60
[table-planner] Use serializable string for ObjectIdentifier in JSON
commit a65c12b7c6ad6ba2b9ba21f0fcb749ddde2fc684
[table-planner] Add Parser to the SerdeContext
> Make ObjectIdentifier json representation simpler
> -
>
> Key: FLINK-25791
> URL: https://issues.apache.org/jira/browse/FLINK-25791
> Project: Flink
> Issue Type: Sub-task
>Reporter: Francesco Guardiani
>Assignee: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Use {{ObjectIdentifier#asSerializableString}} to serialize the object
> identifier, rather than serializing it as object.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25794) Memory pages in LazyMemorySegmentPool should be clear after they are released to MemoryManager
[ https://issues.apache.org/jira/browse/FLINK-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481666#comment-17481666 ] Shammon commented on FLINK-25794: - Thanks [~lzljs3620320], I think I ignored the operations in `MemoryManager`. Then how about add `table.free()` in `BinaryHashTableTest.testBinaryHashBucketAreaNotEnoughMem`? > Memory pages in LazyMemorySegmentPool should be clear after they are released > to MemoryManager > -- > > Key: FLINK-25794 > URL: https://issues.apache.org/jira/browse/FLINK-25794 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Affects Versions: 1.11.6, 1.13.5, 1.14.3 >Reporter: Shammon >Assignee: Shammon >Priority: Major > Labels: pull-request-available > > `LazyMemorySegmentPool` manages memory segments cache for join, agg, sort and > etc. operators. These segments in the cache will be released to > `MemoryManager` after some specify operations such as join operator finishes > to build data in `LazyMemorySegmentPool.cleanCache` method. But these > segments are still in `LazyMemorySegmentPool.cachePages`, it may cause memory > fault if the `MemoryManager` has deallocated these segments -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
flinkbot edited a comment on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1015848205 ## CI report: * 1cc4394ed63adc3729cc13c2ef7aebebdf0d4e87 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30080) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tillrohrmann commented on a change in pull request #284: [FLINK-25775] Add Javascript SDK documentation
tillrohrmann commented on a change in pull request #284:
URL: https://github.com/apache/flink-statefun/pull/284#discussion_r791504850
##
File path: docs/content/docs/sdk/js.md
##
@@ -0,0 +1,267 @@
+---
+title: JavaScript
+weight: 2
+type: docs
+aliases:
+ - /sdk/js.html
+---
+
+
+# JavaScript SDK
+
+Stateful functions are the building blocks of applications; they are atomic
units of isolation, distribution, and persistence.
+As objects, they encapsulate the state of a single entity (e.g., a specific
user, device, or session) and encode its behavior.
+Stateful functions can interact with each other, and external systems, through
message passing.
+
+To get started, add the JavaScript SDK as a dependency to your application.
+
+{{< selectable >}}
+```
+npm install apache-flink-statefun@{{< version >}}
+```
+{{< /selectable >}}
+
+## Defining a Stateful Function
+
+A stateful function is any function that takes a `context` and `message`
parameter.
+In the following example, a `StatefulFunction` maintains a count for every user
+of an application, emitting a customized greeting.
+
+```javascript
+const {messageBuilder, StateFun, Context} = require("apache-flink-statefun");
+
+let statefun = new StateFun();
+
+statefun.bind({
+typename: "com.example.fns/greeter",
+fn(context, message) {
+const name = message.asString();
+let seen = storage.seen || 0;
Review comment:
```suggestion
let seen = context.storage.seen || 0;
```
##
File path: docs/content/docs/sdk/js.md
##
@@ -0,0 +1,267 @@
+---
+title: JavaScript
+weight: 2
+type: docs
+aliases:
+ - /sdk/js.html
+---
+
+
+# JavaScript SDK
+
+Stateful functions are the building blocks of applications; they are atomic
units of isolation, distribution, and persistence.
+As objects, they encapsulate the state of a single entity (e.g., a specific
user, device, or session) and encode its behavior.
+Stateful functions can interact with each other, and external systems, through
message passing.
+
+To get started, add the JavaScript SDK as a dependency to your application.
+
+{{< selectable >}}
+```
+npm install apache-flink-statefun@{{< version >}}
+```
+{{< /selectable >}}
+
+## Defining a Stateful Function
+
+A stateful function is any function that takes a `context` and `message`
parameter.
+In the following example, a `StatefulFunction` maintains a count for every user
+of an application, emitting a customized greeting.
+
+```javascript
+const {messageBuilder, StateFun, Context} = require("apache-flink-statefun");
+
+let statefun = new StateFun();
+
+statefun.bind({
+typename: "com.example.fns/greeter",
+fn(context, message) {
+const name = message.asString();
+let seen = storage.seen || 0;
+seen = seen + 1;
+storage.seen = seen;
Review comment:
```suggestion
context.storage.seen = seen;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
flinkbot edited a comment on pull request #18397: URL: https://github.com/apache/flink/pull/18397#issuecomment-1015848205 ## CI report: * 1cc4394ed63adc3729cc13c2ef7aebebdf0d4e87 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30080) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791509668
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/AbstractZooKeeperHaServices.java
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability.zookeeper;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory;
+import org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointRecoveryFactory;
+import org.apache.flink.runtime.highavailability.AbstractHaServices;
+import org.apache.flink.runtime.highavailability.RunningJobsRegistry;
+import org.apache.flink.runtime.jobmanager.JobGraphStore;
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+import org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths;
+import org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException;
+
+import javax.annotation.Nonnull;
+
+import java.util.concurrent.Executor;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** Abstract ZooKeeper based HA services. */
+public abstract class AbstractZooKeeperHaServices extends AbstractHaServices {
+/** The curator resource to use. */
+private final CuratorFrameworkWithUnhandledErrorListener
curatorFrameworkWrapper;
+
+public AbstractZooKeeperHaServices(
+CuratorFrameworkWithUnhandledErrorListener curatorFrameworkWrapper,
+Executor executor,
+Configuration configuration,
+BlobStoreService blobStoreService) {
+super(configuration, executor, blobStoreService);
+this.curatorFrameworkWrapper = checkNotNull(curatorFrameworkWrapper);
+}
+
+protected final CuratorFramework getCuratorFramework() {
+return curatorFrameworkWrapper.asCuratorFramework();
+}
+
+@Override
+public CheckpointRecoveryFactory createCheckpointRecoveryFactory() throws
Exception {
+return new ZooKeeperCheckpointRecoveryFactory(
+ZooKeeperUtils.useNamespaceAndEnsurePath(
+curatorFrameworkWrapper.asCuratorFramework(),
ZooKeeperUtils.getJobsPath()),
+configuration,
+ioExecutor);
+}
+
+@Override
+public JobGraphStore createJobGraphStore() throws Exception {
+return ZooKeeperUtils.createJobGraphs(
+curatorFrameworkWrapper.asCuratorFramework(), configuration);
+}
+
+@Override
+public RunningJobsRegistry createRunningJobsRegistry() {
+return new ZooKeeperRunningJobsRegistry(
+curatorFrameworkWrapper.asCuratorFramework(), configuration);
+}
+
+@Override
+protected void internalClose() throws Exception {
+curatorFrameworkWrapper.close();
+}
+
+@Override
+protected void internalCleanup() throws Exception {
+cleanupZooKeeperPaths();
+}
+
+@Override
+protected void internalCleanupJobData(JobID jobID) throws Exception {
+deleteZNode(ZooKeeperUtils.getLeaderPathForJob(jobID));
+}
+
+/** Cleans up leftover ZooKeeper paths. */
+private void cleanupZooKeeperPaths() throws Exception {
+deleteOwnedZNode();
+tryDeleteEmptyParentZNodes();
+}
+
+private void deleteOwnedZNode() throws Exception {
+deleteZNode("/");
+}
+
+protected void deleteZNode(String path) throws Exception {
+
ZooKeeperUtils.deleteZNode(curatorFrameworkWrapper.asCuratorFramework(), path);
+}
+
+/**
+ * Tries to delete empty parent znodes.
+ *
+ * IMPORTANT: This method can be removed once all supported ZooKeeper
versions support the
+ * container {@link org.apache.zookeeper.CreateMode}.
+ *
+ * @throws Exception if the deletion fails for other reason than {@link
+ * KeeperException.NotEmptyException}
+ */
+private void tryDeleteEmptyParentZNodes() th
[jira] [Commented] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481671#comment-17481671
]
Chesnay Schepler commented on FLINK-18356:
--
??the maximum memory required is 4G/3.xG for each process. This is also [weird]
since we have limit the heap to 2G??
{{-Xmx}} only controls the heap, and according to your table the heap does not
exceed that.
??we have class leaks??
This seems pretty much confirmed at this point.
If I remember correctly in a heap dump I looked at a while ago that Scala
itself can cache class references.
??One possible hotfix is to not reuse the process first.??
I'd be fine with this as a band-aid for the time being because this is
impairing CI quite significantly, but we do nevertheless need to figure out
what the actual cause is. As is I'd think this leak should also be present in
production.
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
> Attachments: 1234.jpg
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791511623
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperMultipleComponentLeaderElectionHaServices.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability.zookeeper;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.MultipleComponentLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverFactory;
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+
+/**
+ * ZooKeeper HA services that only use a single leader election per process.
+ *
+ *
+ * /flink
+ * +/cluster_id_1/leader/latch
+ * || /resource_manager/connection_info
+ * || /dispatcher/connection_info
+ * || /rest_server/connection_info
+ * || /job-id-1/connection_info
+ * || /job-id-2/connection_info
+ * ||
+ * ||
+ * |+jobgraphs/job-id-1
+ * || /job-id-2
+ * |+jobs/job-id-1/checkpoints/latest
+ * | |/latest-1
+ * | |/latest-2
+ * | | /checkpoint_id_counter
+ *
+ */
+public class ZooKeeperMultipleComponentLeaderElectionHaServices
+extends AbstractZooKeeperHaServices {
+
+private final Object lock = new Object();
+
+private final CuratorFramework leaderNamespacedCuratorFramework;
+
+private final FatalErrorHandler fatalErrorHandler;
+
+@Nullable
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService = null;
+
+public ZooKeeperMultipleComponentLeaderElectionHaServices(
+CuratorFrameworkWithUnhandledErrorListener curatorFrameworkWrapper,
+Configuration config,
+Executor ioExecutor,
+BlobStoreService blobStoreService,
+FatalErrorHandler fatalErrorHandler)
+throws Exception {
+super(curatorFrameworkWrapper, ioExecutor, config, blobStoreService);
+this.leaderNamespacedCuratorFramework =
+ZooKeeperUtils.useNamespaceAndEnsurePath(
+getCuratorFramework(), ZooKeeperUtils.getLeaderPath());
+this.fatalErrorHandler = fatalErrorHandler;
+}
+
+@Override
+protected LeaderElectionService createLeaderElectionService(String
leaderName) {
+final MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService;
+
+synchronized (lock) {
+multipleComponentLeaderElectionService =
getOrInitializeSingleLeaderElectionService();
+}
+
+return new DefaultLeaderElectionService(
+
multipleComponentLeaderElectionService.createDriverFactory(leaderName));
+}
+
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
getOrInitializeSingleLeaderElectionService() {
+if (multipleComponentLeaderElectionService == null) {
+try {
+multipleComponentLeaderElectionService =
+ne
[GitHub] [flink] dmvk commented on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
dmvk commented on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1020976719 Hi @zjureel, thanks for the PR. One question on the approach, why do we need to create a new implementation of the `ScheduledExecutor` instead of reusing the `ComponentMainThreadExecutor`, that already implements this interface? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791514100
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperMultipleComponentLeaderElectionHaServices.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability.zookeeper;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.MultipleComponentLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverFactory;
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+
+/**
+ * ZooKeeper HA services that only use a single leader election per process.
+ *
+ *
+ * /flink
+ * +/cluster_id_1/leader/latch
+ * || /resource_manager/connection_info
+ * || /dispatcher/connection_info
+ * || /rest_server/connection_info
+ * || /job-id-1/connection_info
+ * || /job-id-2/connection_info
+ * ||
+ * ||
+ * |+jobgraphs/job-id-1
+ * || /job-id-2
+ * |+jobs/job-id-1/checkpoints/latest
+ * | |/latest-1
+ * | |/latest-2
+ * | | /checkpoint_id_counter
+ *
+ */
+public class ZooKeeperMultipleComponentLeaderElectionHaServices
+extends AbstractZooKeeperHaServices {
+
+private final Object lock = new Object();
+
+private final CuratorFramework leaderNamespacedCuratorFramework;
+
+private final FatalErrorHandler fatalErrorHandler;
+
+@Nullable
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService = null;
+
+public ZooKeeperMultipleComponentLeaderElectionHaServices(
+CuratorFrameworkWithUnhandledErrorListener curatorFrameworkWrapper,
+Configuration config,
+Executor ioExecutor,
+BlobStoreService blobStoreService,
+FatalErrorHandler fatalErrorHandler)
+throws Exception {
+super(curatorFrameworkWrapper, ioExecutor, config, blobStoreService);
+this.leaderNamespacedCuratorFramework =
+ZooKeeperUtils.useNamespaceAndEnsurePath(
+getCuratorFramework(), ZooKeeperUtils.getLeaderPath());
+this.fatalErrorHandler = fatalErrorHandler;
+}
+
+@Override
+protected LeaderElectionService createLeaderElectionService(String
leaderName) {
+final MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService;
+
+synchronized (lock) {
+multipleComponentLeaderElectionService =
getOrInitializeSingleLeaderElectionService();
+}
+
+return new DefaultLeaderElectionService(
+
multipleComponentLeaderElectionService.createDriverFactory(leaderName));
+}
+
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
getOrInitializeSingleLeaderElectionService() {
+if (multipleComponentLeaderElectionService == null) {
+try {
+multipleComponentLeaderElectionService =
+ne
[GitHub] [flink] zentol merged pull request #17498: [FLINK-14954][rest] Add OpenAPI spec generator
zentol merged pull request #17498: URL: https://github.com/apache/flink/pull/17498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-14954) Publish OpenAPI specification of REST Monitoring API
[ https://issues.apache.org/jira/browse/FLINK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-14954. Resolution: Fixed master: 9ab8dbd2cb25a8e11221071ed20712df2d9bf691 > Publish OpenAPI specification of REST Monitoring API > > > Key: FLINK-14954 > URL: https://issues.apache.org/jira/browse/FLINK-14954 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / REST >Reporter: Michaël Melchiore >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Hello, > Flink provides a very helpful REST Monitoring API. > OpenAPI is convenient standard to generate clients in a variety of language > for REST API documented according to their specification. In this case, > clients would be helpful to automate management of Flink clusters. > Currently, there is no "official" OpenAPI specification of Flink REST > Monitoring API. [Some|https://github.com/nextbreakpoint/flink-client] have > written by users, but their consistency across Flink releases is uncertain. > I think it would be beneficial to have an OpenAPI specification provided and > maintained by the Flink project. > > Kind regards, > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14954) Publish OpenAPI specification of REST Monitoring API
[ https://issues.apache.org/jira/browse/FLINK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-14954: - Issue Type: New Feature (was: Improvement) > Publish OpenAPI specification of REST Monitoring API > > > Key: FLINK-14954 > URL: https://issues.apache.org/jira/browse/FLINK-14954 > Project: Flink > Issue Type: New Feature > Components: Documentation, Runtime / REST >Reporter: Michaël Melchiore >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Hello, > Flink provides a very helpful REST Monitoring API. > OpenAPI is convenient standard to generate clients in a variety of language > for REST API documented according to their specification. In this case, > clients would be helpful to automate management of Flink clusters. > Currently, there is no "official" OpenAPI specification of Flink REST > Monitoring API. [Some|https://github.com/nextbreakpoint/flink-client] have > written by users, but their consistency across Flink releases is uncertain. > I think it would be beneficial to have an OpenAPI specification provided and > maintained by the Flink project. > > Kind regards, > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18476: [FLINK-25726][streaming] Add Global Committer as post commit topology
flinkbot edited a comment on pull request #18476: URL: https://github.com/apache/flink/pull/18476#issuecomment-1020132926 ## CI report: * 9c24ac788a7b5c48ce690a43212bcd25d68cf667 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30053) * ff95aa4c4b60ccec3a55723df0b42eb8890a6cc6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25771) CassandraConnectorITCase.testRetrialAndDropTables fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481676#comment-17481676
]
Etienne Chauchot commented on FLINK-25771:
--
[~trohrmann] thanks for pointing out ! I was busy on another ASF project, I'm
taking a look at this ticket now, can you assign me ?
> CassandraConnectorITCase.testRetrialAndDropTables fails on AZP
> --
>
> Key: FLINK-25771
> URL: https://issues.apache.org/jira/browse/FLINK-25771
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Cassandra
>Affects Versions: 1.13.5
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
>
> The test {{CassandraConnectorITCase.testRetrialAndDropTables}} fails on AZP
> with
> {code}
> Jan 23 01:02:52 com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried for query failed (tried: /172.17.0.1:59220
> (com.datastax.driver.core.exceptions.OperationTimedOutException:
> [/172.17.0.1] Timed out waiting for server response))
> Jan 23 01:02:52 at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> Jan 23 01:02:52 at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
> Jan 23 01:02:52 at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> Jan 23 01:02:52 at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
> Jan 23 01:02:52 at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
> Jan 23 01:02:52 at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> Jan 23 01:02:52 at
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.testRetrialAndDropTables(CassandraConnectorITCase.java:554)
> Jan 23 01:02:52 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jan 23 01:02:52 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jan 23 01:02:52 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jan 23 01:02:52 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 23 01:02:52 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Jan 23 01:02:52 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jan 23 01:02:52 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Jan 23 01:02:52 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jan 23 01:02:52 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Jan 23 01:02:52 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Jan 23 01:02:52 at
> org.apache.flink.testutils.junit.RetryRule$RetryOnExceptionStatement.evaluate(RetryRule.java:196)
> Jan 23 01:02:52 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Jan 23 01:02:52 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Jan 23 01:02:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Jan 23 01:02:52 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Jan 23 01:02:52 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 23 01:02:52 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Jan 23 01:02:52 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Jan 23 01:02:52 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:30)
> Jan 23 01:02:52 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 23 01:02:52 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 23 01:02:52 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 23 01:02:52 at org.ju
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485: URL: https://github.com/apache/flink/pull/17485#discussion_r791518309 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java ## @@ -96,13 +106,17 @@ /** The prefix of the completed checkpoint file. */ public static final String HA_STORAGE_COMPLETED_CHECKPOINT = "completedCheckpoint"; -private static final String RESOURCE_MANAGER_LEADER = "/resource_manager"; +private static final String RESOURCE_MANAGER_LEADER = "resource_manager"; -private static final String DISPATCHER_LEADER = "/dispatcher"; +private static final String DISPATCHER_LEADER = "dispatcher"; private static final String LEADER_NODE = "/leader"; Review comment: True. This is a nice unification. Will apply it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
wanglijie95 commented on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1020983443 Squashed commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) * ddb31ae67df56d718052cda835c5dc64147ab928 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18476: [FLINK-25726][streaming] Add Global Committer as post commit topology
flinkbot edited a comment on pull request #18476: URL: https://github.com/apache/flink/pull/18476#issuecomment-1020132926 ## CI report: * 9c24ac788a7b5c48ce690a43212bcd25d68cf667 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30053) * ff95aa4c4b60ccec3a55723df0b42eb8890a6cc6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30128) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #18489: [FLINK-25790][flink-gs-fs-hadoop] Support authentication via core-site.xml in GCS FileSystem plugin
xintongsong commented on a change in pull request #18489:
URL: https://github.com/apache/flink/pull/18489#discussion_r791523758
##
File path:
flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystem.java
##
@@ -39,17 +45,67 @@
GSFileSystem(GoogleHadoopFileSystem googleHadoopFileSystem,
GSFileSystemOptions options) {
super(Preconditions.checkNotNull(googleHadoopFileSystem));
-LOGGER.info("Creating GSFileSystem with options {}", options);
-
this.options = Preconditions.checkNotNull(options);
+LOGGER.info("Creating GSFileSystem with options {}", options);
}
@Override
-public RecoverableWriter createRecoverableWriter() {
-LOGGER.info("Creating recoverable writer with options {}", options);
+public RecoverableWriter createRecoverableWriter() throws IOException {
+
+// follow the same rules as for the Hadoop connector, i.e.
+// 1) only use service credentials at all if Hadoop
+// "google.cloud.auth.service.account.enable" is true (default: true)
+// 2) use GOOGLE_APPLICATION_CREDENTIALS as location of credentials,
if supplied
+// 3) use Hadoop "google.cloud.auth.service.account.json.keyfile" as
location of
+// credentials, if supplied
+// 4) use no credentials
Review comment:
Looking into this a bit more, I think it's probably fine as is.
The module currently consists of two parts leveraging different underlying
libaraies: the FileSystem that uses `gcs-connector`, and the RecoverableWriter
that uses `google-cloud-storage`. Hadoop configurations
(`core-site/default.xml`) can be applied directly on `gcs-connector` but not
`google-cloud-storage`.
In that sense, it makes sense to me that
RecoverableWriter/`google-cloud-storage` only support selective Hadoop
configurations. As a first step, the supported configurations includes only
`account.enable` and `account.json.keyfile`. We can add more if new demands
emerge later.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485: URL: https://github.com/apache/flink/pull/17485#discussion_r791524500 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java ## @@ -96,13 +106,17 @@ /** The prefix of the completed checkpoint file. */ public static final String HA_STORAGE_COMPLETED_CHECKPOINT = "completedCheckpoint"; -private static final String RESOURCE_MANAGER_LEADER = "/resource_manager"; +private static final String RESOURCE_MANAGER_LEADER = "resource_manager"; -private static final String DISPATCHER_LEADER = "/dispatcher"; +private static final String DISPATCHER_LEADER = "dispatcher"; private static final String LEADER_NODE = "/leader"; -private static final String REST_SERVER_LEADER = "/rest_server"; +private static final String REST_SERVER_LEADER = "rest_server"; + +private static final String LEADER_LATCH_NODE = "/latch"; Review comment: Yes. Will update it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
flinkbot edited a comment on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1014266348 ## CI report: * 320757eaf30cae2a2b95c06e559c37e068120ef4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30042) * 87627f9924e44eb40028822fc427363ea6d845a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791527216
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperMultipleComponentLeaderElectionHaServices.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability.zookeeper;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.MultipleComponentLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverFactory;
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+
+/**
+ * ZooKeeper HA services that only use a single leader election per process.
+ *
+ *
+ * /flink
+ * +/cluster_id_1/leader/latch
+ * || /resource_manager/connection_info
+ * || /dispatcher/connection_info
+ * || /rest_server/connection_info
+ * || /job-id-1/connection_info
+ * || /job-id-2/connection_info
+ * ||
+ * ||
+ * |+jobgraphs/job-id-1
+ * || /job-id-2
+ * |+jobs/job-id-1/checkpoints/latest
+ * | |/latest-1
+ * | |/latest-2
+ * | | /checkpoint_id_counter
+ *
+ */
+public class ZooKeeperMultipleComponentLeaderElectionHaServices
+extends AbstractZooKeeperHaServices {
+
+private final Object lock = new Object();
+
+private final CuratorFramework leaderNamespacedCuratorFramework;
+
+private final FatalErrorHandler fatalErrorHandler;
+
+@Nullable
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService = null;
+
+public ZooKeeperMultipleComponentLeaderElectionHaServices(
+CuratorFrameworkWithUnhandledErrorListener curatorFrameworkWrapper,
+Configuration config,
+Executor ioExecutor,
+BlobStoreService blobStoreService,
+FatalErrorHandler fatalErrorHandler)
+throws Exception {
+super(curatorFrameworkWrapper, ioExecutor, config, blobStoreService);
+this.leaderNamespacedCuratorFramework =
+ZooKeeperUtils.useNamespaceAndEnsurePath(
+getCuratorFramework(), ZooKeeperUtils.getLeaderPath());
+this.fatalErrorHandler = fatalErrorHandler;
+}
+
+@Override
+protected LeaderElectionService createLeaderElectionService(String
leaderName) {
+final MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService;
+
+synchronized (lock) {
+multipleComponentLeaderElectionService =
getOrInitializeSingleLeaderElectionService();
+}
+
+return new DefaultLeaderElectionService(
+
multipleComponentLeaderElectionService.createDriverFactory(leaderName));
+}
+
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
getOrInitializeSingleLeaderElectionService() {
+if (multipleComponentLeaderElectionService == null) {
+try {
+multipleComponentLeaderElectionService =
+ne
[GitHub] [flink] KarmaGYZ commented on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
KarmaGYZ commented on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1020991108 @dmvk That's what we did in #18007. In this PR, we try to find which method has some specific logic and cannot use the new `schedule`. However, once we ensure that this PR is not related to FLINK-18356, we can return to the previous implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791529865
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperMultipleComponentLeaderElectionHaServices.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability.zookeeper;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.DefaultMultipleComponentLeaderElectionService;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.MultipleComponentLeaderElectionService;
+import
org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverFactory;
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+
+/**
+ * ZooKeeper HA services that only use a single leader election per process.
+ *
+ *
+ * /flink
+ * +/cluster_id_1/leader/latch
+ * || /resource_manager/connection_info
+ * || /dispatcher/connection_info
+ * || /rest_server/connection_info
+ * || /job-id-1/connection_info
+ * || /job-id-2/connection_info
+ * ||
+ * ||
+ * |+jobgraphs/job-id-1
+ * || /job-id-2
+ * |+jobs/job-id-1/checkpoints/latest
+ * | |/latest-1
+ * | |/latest-2
+ * | | /checkpoint_id_counter
+ *
+ */
+public class ZooKeeperMultipleComponentLeaderElectionHaServices
+extends AbstractZooKeeperHaServices {
+
+private final Object lock = new Object();
+
+private final CuratorFramework leaderNamespacedCuratorFramework;
+
+private final FatalErrorHandler fatalErrorHandler;
+
+@Nullable
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService = null;
+
+public ZooKeeperMultipleComponentLeaderElectionHaServices(
+CuratorFrameworkWithUnhandledErrorListener curatorFrameworkWrapper,
+Configuration config,
+Executor ioExecutor,
+BlobStoreService blobStoreService,
+FatalErrorHandler fatalErrorHandler)
+throws Exception {
+super(curatorFrameworkWrapper, ioExecutor, config, blobStoreService);
+this.leaderNamespacedCuratorFramework =
+ZooKeeperUtils.useNamespaceAndEnsurePath(
+getCuratorFramework(), ZooKeeperUtils.getLeaderPath());
+this.fatalErrorHandler = fatalErrorHandler;
+}
+
+@Override
+protected LeaderElectionService createLeaderElectionService(String
leaderName) {
+final MultipleComponentLeaderElectionService
multipleComponentLeaderElectionService;
+
+synchronized (lock) {
+multipleComponentLeaderElectionService =
getOrInitializeSingleLeaderElectionService();
+}
+
+return new DefaultLeaderElectionService(
+
multipleComponentLeaderElectionService.createDriverFactory(leaderName));
+}
+
+@GuardedBy("lock")
+private MultipleComponentLeaderElectionService
getOrInitializeSingleLeaderElectionService() {
+if (multipleComponentLeaderElectionService == null) {
+try {
+multipleComponentLeaderElectionService =
+ne
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485: URL: https://github.com/apache/flink/pull/17485#discussion_r791530279 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java ## @@ -96,13 +106,17 @@ /** The prefix of the completed checkpoint file. */ public static final String HA_STORAGE_COMPLETED_CHECKPOINT = "completedCheckpoint"; -private static final String RESOURCE_MANAGER_LEADER = "/resource_manager"; +private static final String RESOURCE_MANAGER_LEADER = "resource_manager"; Review comment: This has been addressed in a previous fixup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) * ddb31ae67df56d718052cda835c5dc64147ab928 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18376: [FLINK-25668][runtime] Support calcuate network memory for dynamic graph
flinkbot edited a comment on pull request #18376: URL: https://github.com/apache/flink/pull/18376#issuecomment-1014266348 ## CI report: * 320757eaf30cae2a2b95c06e559c37e068120ef4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30042) * 87627f9924e44eb40028822fc427363ea6d845a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30130) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791531251
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java
##
@@ -137,17 +155,37 @@ public static String getCheckpointIdCounterPath() {
return "/checkpoint_id_counter";
}
+public static String getLeaderPath() {
+return LEADER_NODE;
+}
+
+public static String getDispatcherNode() {
+return DISPATCHER_LEADER;
+}
+
+public static String getResourceManagerNode() {
+return RESOURCE_MANAGER_LEADER;
+}
+
+public static String getRestServerNode() {
+return REST_SERVER_LEADER;
+}
+
+public static String getLeaderLatchNode() {
+return LEADER_LATCH_NODE;
+}
+
private static String getLeaderPath(String suffix) {
return generateZookeeperPath(LEADER_NODE, suffix);
}
@Nonnull
Review comment:
True, but I haven't touched this code in this PR. Can be a follow-up
task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18483: [FLINK-24041][connectors] Removed public setter for elementConverter in As…
flinkbot edited a comment on pull request #18483: URL: https://github.com/apache/flink/pull/18483#issuecomment-1020286441 ## CI report: * ddf6b397383be3096f061ee5f292c6bf0b204fa6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30084) * 2b460acc3ba95ca362aea9d04f67de1a9e00df9b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 closed pull request #18397: [FLINK-25702][Kafka] Use the configure feature provided by the kafka Serializer/Deserializer.
lindong28 closed pull request #18397: URL: https://github.com/apache/flink/pull/18397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791532138
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java
##
@@ -555,6 +668,10 @@ public static String generateZookeeperPath(String root,
String child) {
return result;
}
+public static String makeZooKeeperPath(String path) {
Review comment:
True, I will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dannycranmer commented on pull request #18488: [FLINK-24041][connector] Fixing reversed insertion of failed requests
dannycranmer commented on pull request #18488: URL: https://github.com/apache/flink/pull/18488#issuecomment-1020996381 LGTM, merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dannycranmer merged pull request #18488: [FLINK-24041][connector] Fixing reversed insertion of failed requests
dannycranmer merged pull request #18488: URL: https://github.com/apache/flink/pull/18488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791534467
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/ZooKeeperMultipleComponentLeaderElectionDriver.java
##
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.leaderelection;
+
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.concurrent.Executors;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.ChildData;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCacheSelector;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.leader.LeaderLatch;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.leader.LeaderLatchListener;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionState;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/** ZooKeeper based {@link MultipleComponentLeaderElectionDriver}
implementation. */
+public class ZooKeeperMultipleComponentLeaderElectionDriver
+implements MultipleComponentLeaderElectionDriver, LeaderLatchListener {
+
+private static final Logger LOG =
+
LoggerFactory.getLogger(ZooKeeperMultipleComponentLeaderElectionDriver.class);
+
+private final CuratorFramework curatorFramework;
+
+private final String leaderContenderDescription;
+
+private final MultipleComponentLeaderElectionDriver.Listener
leaderElectionListener;
+
+private final LeaderLatch leaderLatch;
+
+private final TreeCache treeCache;
+
+private final ConnectionStateListener listener =
+(client, newState) -> handleStateChange(newState);
+
+private AtomicBoolean running = new AtomicBoolean(true);
+
+public ZooKeeperMultipleComponentLeaderElectionDriver(
+CuratorFramework curatorFramework,
+String leaderContenderDescription,
+MultipleComponentLeaderElectionDriver.Listener
leaderElectionListener)
+throws Exception {
+this.curatorFramework = curatorFramework;
+this.leaderContenderDescription = leaderContenderDescription;
+this.leaderElectionListener = leaderElectionListener;
Review comment:
I am a bit torn here. The general Flink guideline says that arguments
that are not annotated with `@Nullable` can be assumed to be non-null. Hence, a
null check should not strictly be necessary. On the other hand, failing fast in
case of a NPE is always a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25794) Memory pages in LazyMemorySegmentPool should be clear after they are released to MemoryManager
[ https://issues.apache.org/jira/browse/FLINK-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481690#comment-17481690 ] Jingsong Lee commented on FLINK-25794: -- [~zjureel] What do you mean? Memory leak in `BinaryHashTableTest.testBinaryHashBucketAreaNotEnoughMem`? > Memory pages in LazyMemorySegmentPool should be clear after they are released > to MemoryManager > -- > > Key: FLINK-25794 > URL: https://issues.apache.org/jira/browse/FLINK-25794 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Affects Versions: 1.11.6, 1.13.5, 1.14.3 >Reporter: Shammon >Assignee: Shammon >Priority: Major > Labels: pull-request-available > > `LazyMemorySegmentPool` manages memory segments cache for join, agg, sort and > etc. operators. These segments in the cache will be released to > `MemoryManager` after some specify operations such as join operator finishes > to build data in `LazyMemorySegmentPool.cleanCache` method. But these > segments are still in `LazyMemorySegmentPool.cachePages`, it may cause memory > fault if the `MemoryManager` has deallocated these segments -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #14: [FLINK-25803] Implement partition and bucket filter in FileStoreScanImpl
tsreaper commented on a change in pull request #14:
URL: https://github.com/apache/flink-table-store/pull/14#discussion_r791535456
##
File path:
flink-table-store-core/src/test/java/org/apache/flink/table/store/file/operation/FileStoreCommitTestBase.java
##
@@ -71,7 +67,7 @@ public void beforeEach() throws IOException {
root.getFileSystem().mkdirs(new Path(root + "/snapshot"));
}
-protected abstract String getSchema();
+protected abstract String getScheme();
Review comment:
Scheme. This is the file system scheme, not table schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18483: [FLINK-24041][connectors] Removed public setter for elementConverter in As…
flinkbot edited a comment on pull request #18483: URL: https://github.com/apache/flink/pull/18483#issuecomment-1020286441 ## CI report: * ddf6b397383be3096f061ee5f292c6bf0b204fa6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30084) * 2b460acc3ba95ca362aea9d04f67de1a9e00df9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30131) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #14: [FLINK-25803] Implement partition and bucket filter in FileStoreScanImpl
tsreaper commented on a change in pull request #14:
URL: https://github.com/apache/flink-table-store/pull/14#discussion_r791536162
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/utils/FileStorePathFactory.java
##
@@ -106,10 +115,12 @@ public SstPathFactory createSstPathFactory(BinaryRowData
partition, int bucket)
}
public String getPartitionString(BinaryRowData partition) {
-return PartitionPathUtils.generatePartitionPath(
-partitionComputer.generatePartValues(
-Preconditions.checkNotNull(
-partition, "Partition row data is null. This
is unexpected.")));
+synchronized (partitionComputer) {
Review comment:
Because we have to make sure that this class is thread safe. Partition
computer is not thread safe.
Another option is to create a read-only path factory which only supports a
limited number of methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #14: [FLINK-25803] Implement partition and bucket filter in FileStoreScanImpl
tsreaper commented on a change in pull request #14:
URL: https://github.com/apache/flink-table-store/pull/14#discussion_r791536943
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/FileStoreScanImpl.java
##
@@ -109,34 +176,72 @@ public Long snapshotId() {
}
private List scan() {
-Map map = new
LinkedHashMap<>();
-for (ManifestFileMeta manifest : manifests) {
-// TODO read each manifest file concurrently
-for (ManifestEntry entry : manifestFile.read(manifest.fileName()))
{
-ManifestEntry.Identifier identifier = entry.identifier();
-switch (entry.kind()) {
-case ADD:
-Preconditions.checkState(
-!map.containsKey(identifier),
-"Trying to add file %s which is already added.
"
-+ "Manifest might be corrupted.",
-identifier);
-map.put(identifier, entry);
-break;
-case DELETE:
-Preconditions.checkState(
-map.containsKey(identifier),
-"Trying to delete file %s which is not
previously added. "
-+ "Manifest might be corrupted.",
-identifier);
-map.remove(identifier);
-break;
-default:
-throw new UnsupportedOperationException(
-"Unknown value kind " + entry.kind().name());
-}
+List entries;
+try {
+entries =
+new ForkJoinPool()
Review comment:
If we want to use `parallelStream` we need `ForkJoinPool`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17485: [FLINK-24038] Add support for single leader election per JobManager process
tillrohrmann commented on a change in pull request #17485:
URL: https://github.com/apache/flink/pull/17485#discussion_r791540131
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/ZooKeeperMultipleComponentLeaderElectionDriver.java
##
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.leaderelection;
+
+import org.apache.flink.runtime.util.ZooKeeperUtils;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.concurrent.Executors;
+
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.ChildData;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCache;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.TreeCacheSelector;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.leader.LeaderLatch;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.leader.LeaderLatchListener;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionState;
+import
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/** ZooKeeper based {@link MultipleComponentLeaderElectionDriver}
implementation. */
+public class ZooKeeperMultipleComponentLeaderElectionDriver
+implements MultipleComponentLeaderElectionDriver, LeaderLatchListener {
+
+private static final Logger LOG =
+
LoggerFactory.getLogger(ZooKeeperMultipleComponentLeaderElectionDriver.class);
+
+private final CuratorFramework curatorFramework;
+
+private final String leaderContenderDescription;
+
+private final MultipleComponentLeaderElectionDriver.Listener
leaderElectionListener;
+
+private final LeaderLatch leaderLatch;
+
+private final TreeCache treeCache;
+
+private final ConnectionStateListener listener =
+(client, newState) -> handleStateChange(newState);
+
+private AtomicBoolean running = new AtomicBoolean(true);
+
+public ZooKeeperMultipleComponentLeaderElectionDriver(
+CuratorFramework curatorFramework,
+String leaderContenderDescription,
+MultipleComponentLeaderElectionDriver.Listener
leaderElectionListener)
+throws Exception {
+this.curatorFramework = curatorFramework;
+this.leaderContenderDescription = leaderContenderDescription;
+this.leaderElectionListener = leaderElectionListener;
+
+this.leaderLatch = new LeaderLatch(curatorFramework,
ZooKeeperUtils.getLeaderLatchNode());
+this.treeCache =
+TreeCache.newBuilder(curatorFramework, "/")
+.setCacheData(true)
+.setCreateParentNodes(false)
+.setSelector(
+new
ZooKeeperMultipleComponentLeaderElectionDriver
+.ConnectionInfoNodeSelector())
+.setExecutor(Executors.newDirectExecutorService())
+.build();
+treeCache
+.getListenable()
+.addListener(
+(client, event) -> {
+switch (event.getType()) {
+case NODE_ADDED:
+case NODE_REMOVED:
+case NODE_UPDATED:
+if (event.getData() != null) {
+
handleChangedLeaderInformation(event.getData());
+}
+}
+});
+
+leaderLatch.addListener(this);
+curatorFramework.getConnectionStateListenable().addListener(listener);
+leaderLatch.start();
+treeCache.start();
+}
+
+
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) * ddb31ae67df56d718052cda835c5dc64147ab928 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * 9acb58bc6e84f5825f6b21cff5e03343379ef132 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30093) * 40baec2c2a17e2d042fd7fb3b0bb925f10a437ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30120) * 56fee29c28007e3ba03d1f14ed7ef12a38a12b0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30126) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shouzuo1 commented on pull request #18358: [FLINK-25651][docs] Update kafka.md
shouzuo1 commented on pull request #18358: URL: https://github.com/apache/flink/pull/18358#issuecomment-1021003850 thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shouzuo1 removed a comment on pull request #18358: [FLINK-25651][docs] Update kafka.md
shouzuo1 removed a comment on pull request #18358: URL: https://github.com/apache/flink/pull/18358#issuecomment-1021003850 thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17481697#comment-17481697
]
Yangze Guo commented on FLINK-18356:
Thanks for the investigation [~gaoyunhaii]. I wonder if I can dismiss this
issue as unrelated to Flink-25085? It seems that any change that affects the
table tests can compound this issue.
cc [~twalthr] WDYT?
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
> Attachments: 1234.jpg
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25804) Loading and running connector code use separated ClassLoader.
Ada Wong created FLINK-25804:
Summary: Loading and running connector code use separated
ClassLoader.
Key: FLINK-25804
URL: https://issues.apache.org/jira/browse/FLINK-25804
Project: Flink
Issue Type: New Feature
Components: API / Core, Connectors / Common, Table SQL / Runtime
Affects Versions: 1.14.3
Reporter: Ada Wong
When we use multiple connectors could have class conflicts. This class conflict
can not be solved by shade.
The following is example code.
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;
{code:java}
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
Inspird by PulginManager, PluginFileSystemFactory and
ClassLoaderFixingFileSystem class.
Could we create many ClassLoaderFixing* class to avoid class conflict. Such as
ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
ClassLoaderFixingSinkFunction.
If we use ClassLoader fixing, each call SinkFunction#invoke will switch
classloader by Thread#currentThread()#setContextClassLoader(). Does
setContextClassLoader() has heavy overhead of setContextClassLoader()?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25804) Loading and running connector code use separated ClassLoader.
[
https://issues.apache.org/jira/browse/FLINK-25804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ada Wong updated FLINK-25804:
-
Priority: Blocker (was: Major)
> Loading and running connector code use separated ClassLoader.
> -
>
> Key: FLINK-25804
> URL: https://issues.apache.org/jira/browse/FLINK-25804
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common, Table SQL / Runtime
>Affects Versions: 1.14.3
>Reporter: Ada Wong
>Priority: Blocker
>
> When we use multiple connectors could have class conflicts. This class
> conflict can not be solved by shade.
> The following is example code.
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;
>
> {code:java}
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
>
> Inspird by PulginManager, PluginFileSystemFactory and
> ClassLoaderFixingFileSystem class.
> Could we create many ClassLoaderFixing* class to avoid class conflict. Such
> as ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
> ClassLoaderFixingSinkFunction.
> If we use ClassLoader fixing, each call SinkFunction#invoke will switch
> classloader by Thread#currentThread()#setContextClassLoader(). Does
> setContextClassLoader() has heavy overhead of setContextClassLoader()?
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25804) Loading and running connector code use separated ClassLoader.
[
https://issues.apache.org/jira/browse/FLINK-25804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ada Wong updated FLINK-25804:
-
Issue Type: Bug (was: New Feature)
> Loading and running connector code use separated ClassLoader.
> -
>
> Key: FLINK-25804
> URL: https://issues.apache.org/jira/browse/FLINK-25804
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common, Table SQL / Runtime
>Affects Versions: 1.14.3
>Reporter: Ada Wong
>Priority: Major
>
> When we use multiple connectors could have class conflicts. This class
> conflict can not be solved by shade.
> The following is example code.
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;
>
> {code:java}
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
>
> Inspird by PulginManager, PluginFileSystemFactory and
> ClassLoaderFixingFileSystem class.
> Could we create many ClassLoaderFixing* class to avoid class conflict. Such
> as ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
> ClassLoaderFixingSinkFunction.
> If we use ClassLoader fixing, each call SinkFunction#invoke will switch
> classloader by Thread#currentThread()#setContextClassLoader(). Does
> setContextClassLoader() has heavy overhead of setContextClassLoader()?
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25804) Loading and running connector code use separated ClassLoader.
[
https://issues.apache.org/jira/browse/FLINK-25804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ada Wong updated FLINK-25804:
-
Description:
When we use multiple connectors could have class conflicts. This class conflict
can not be solved by shade.
The following is example code.
{code:java}
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
Inspird by PulginManager, PluginFileSystemFactory and
ClassLoaderFixingFileSystem class.
Could we create many ClassLoaderFixing* class to avoid class conflict. Such as
ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
ClassLoaderFixingSinkFunction.
If we use ClassLoader fixing, each call SinkFunction#invoke will switch
classloader by Thread#currentThread()#setContextClassLoader(). Does
setContextClassLoader() has heavy overhead of setContextClassLoader()?
was:
When we use multiple connectors could have class conflicts. This class conflict
can not be solved by shade.
The following is example code.
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;
{code:java}
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
Inspird by PulginManager, PluginFileSystemFactory and
ClassLoaderFixingFileSystem class.
Could we create many ClassLoaderFixing* class to avoid class conflict. Such as
ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
ClassLoaderFixingSinkFunction.
If we use ClassLoader fixing, each call SinkFunction#invoke will switch
classloader by Thread#currentThread()#setContextClassLoader(). Does
setContextClassLoader() has heavy overhead of setContextClassLoader()?
> Loading and running connector code use separated ClassLoader.
> -
>
> Key: FLINK-25804
> URL: https://issues.apache.org/jira/browse/FLINK-25804
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common, Table SQL / Runtime
>Affects Versions: 1.14.3
>Reporter: Ada Wong
>Priority: Blocker
>
> When we use multiple connectors could have class conflicts. This class
> conflict can not be solved by shade.
> The following is example code.
> {code:java}
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
>
> Inspird by PulginManager, PluginFileSystemFactory and
> ClassLoaderFixingFileSystem class.
> Could we create many ClassLoaderFixing* class to avoid class conflict. Such
> as ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
> ClassLoaderFixingSinkFunction.
> If we use ClassLoader fixing, each call SinkFunction#invoke will swi
[jira] [Updated] (FLINK-25804) Loading and running connector code use separated ClassLoader.
[
https://issues.apache.org/jira/browse/FLINK-25804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ada Wong updated FLINK-25804:
-
Description:
When we use multiple connectors could have class conflicts. This class conflict
can not be solved by shade.
The following is example code.
{code:java}
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
Inspired by PluginManager, PluginFileSystemFactory and
ClassLoaderFixingFileSystem class.
Could we create many ClassLoaderFixing* class to avoid class conflict. Such as
ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
ClassLoaderFixingSinkFunction.
If we use ClassLoader fixing, each call SinkFunction#invoke will switch
ClassLoader by Thread#currentThread()#setContextClassLoader(). Does
setContextClassLoader() has heavy overhead of setContextClassLoader()?
was:
When we use multiple connectors could have class conflicts. This class conflict
can not be solved by shade.
The following is example code.
{code:java}
CREATE TABLE es6 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-6',
'hosts' = 'http://localhost:9200',
'index' = 'users',
'document-type' = 'foo'
);
CREATE TABLE es7 (
user_id STRING,
user_name STRING,
PRIMARYKEY (user_id) NOT ENFORCED
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'users'
);
CREATE TABLE ods (
user_id STRING,
user_name STRING
) WITH (
'connector' = 'datagen'
);
INSERT INTO es6 SELECT user_id, user_name FROM ods;
INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
Inspird by PulginManager, PluginFileSystemFactory and
ClassLoaderFixingFileSystem class.
Could we create many ClassLoaderFixing* class to avoid class conflict. Such as
ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
ClassLoaderFixingSinkFunction.
If we use ClassLoader fixing, each call SinkFunction#invoke will switch
classloader by Thread#currentThread()#setContextClassLoader(). Does
setContextClassLoader() has heavy overhead of setContextClassLoader()?
> Loading and running connector code use separated ClassLoader.
> -
>
> Key: FLINK-25804
> URL: https://issues.apache.org/jira/browse/FLINK-25804
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common, Table SQL / Runtime
>Affects Versions: 1.14.3
>Reporter: Ada Wong
>Priority: Blocker
>
> When we use multiple connectors could have class conflicts. This class
> conflict can not be solved by shade.
> The following is example code.
> {code:java}
> CREATE TABLE es6 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-6',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users',
> 'document-type' = 'foo'
> );
> CREATE TABLE es7 (
> user_id STRING,
> user_name STRING,
> PRIMARYKEY (user_id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'http://localhost:9200',
> 'index' = 'users'
> );
> CREATE TABLE ods (
> user_id STRING,
> user_name STRING
> ) WITH (
> 'connector' = 'datagen'
> );
> INSERT INTO es6 SELECT user_id, user_name FROM ods;
> INSERT INTO es7 SELECT user_id, user_name FROM ods;{code}
>
> Inspired by PluginManager, PluginFileSystemFactory and
> ClassLoaderFixingFileSystem class.
> Could we create many ClassLoaderFixing* class to avoid class conflict. Such
> as ClassLoaderFixingDynamicTableFactory, ClassLoaderFixingSink or
> ClassLoaderFixingSinkFunction.
> If we use ClassLoader fixing, each call SinkFunction#invoke will switch
> ClassLoader by Thread#currentThread()#setContextClassLoader(). Does
> setContextClassLoader() has heavy overhead of setContextClassLoader()?
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18333: [FLINK-25220][test] Write an architectural rule for all IT cases w.r.t. the MiniCluster
flinkbot edited a comment on pull request #18333: URL: https://github.com/apache/flink/pull/18333#issuecomment-1010381163 ## CI report: * 454378c5d0aecc36ae88a99f1e1d936405226a22 UNKNOWN * cc0ef7b3c9f391a389117dc33107cd9f3d77d708 UNKNOWN * 96c787f1dd4b373c00d1acb03b918c823a0acbee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30072) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz reassigned FLINK-18356:
Assignee: (was: Dawid Wysakowicz)
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
> Attachments: 1234.jpg
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18409: [hotfix][dist] Fix that flink-gs-fs-hadoop not included in flink-dist
flinkbot edited a comment on pull request #18409: URL: https://github.com/apache/flink/pull/18409#issuecomment-1017132626 ## CI report: * 74c02f4b21ad4488b0466a748114e26f15495c4e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29991) * 19e7922b5364c22ed9a4675154778f07fd1ebf34 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30098) * d105000a8f900450acefdcd0102331a2aaa72e47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30105) * c41dcd5aaf3db3d30cdcef448dcede88fa04713b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
