Re: [PR] [FLINK-34660][checkpoint] Avoid randomizing changelog-related config when it's pre-defined [flink]

2024-03-12 Thread via GitHub
flinkbot commented on PR #24488: URL: https://github.com/apache/flink/pull/24488#issuecomment-1993657819 ## CI report: * 740730fcf87de7999e8f56b143bdc97103c2a77e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

Re: [PR] [FLINK-32076][checkpoint] Introduce file pool to reuse files [flink]

2024-03-12 Thread via GitHub
masteryhx commented on PR #24418: URL: https://github.com/apache/flink/pull/24418#issuecomment-1993656300 The failed case caused by [FLINK-34660](https://issues.apache.org/jira/browse/FLINK-34660) which is not related to this PR. I also submitted a pr to resolved this. -- This is an a

[jira] [Updated] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError

2024-03-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34660: --- Labels: pull-request-available (was: ) > AutoRescalingITCase#testCheckpointRescalingInKeyed

[PR] [FLINK-34660][checkpoint] Avoid randomizing changelog-related config when it's pre-defined [flink]

2024-03-12 Thread via GitHub
masteryhx opened a new pull request, #24488: URL: https://github.com/apache/flink/pull/24488 ## What is the purpose of the change AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError: ``` expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000

[jira] [Created] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError

2024-03-12 Thread Hangxiang Yu (Jira)
Hangxiang Yu created FLINK-34660: Summary: AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError Key: FLINK-34660 URL: https://issues.apache.org/jira/browse/FLINK-34660 Project: Flink

[jira] [Commented] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-12 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825913#comment-17825913 ] Ming Li commented on FLINK-33798: - [~fanrui], good idea, I will back port to 1.17 , 1.18

Re: [PR] [FLINK-34543][datastream]Support Full Partition Processing On Non-keyed DataStream [flink]

2024-03-12 Thread via GitHub
reswqa commented on code in PR #24398: URL: https://github.com/apache/flink/pull/24398#discussion_r1522571183 ## flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java: ## @@ -128,6 +128,24 @@ public class ExecutionOptions {

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
LadyForest commented on PR #24390: URL: https://github.com/apache/flink/pull/24390#issuecomment-1993628470 Hi @XComp, I plan to merge PR since the CI passed unless you have any other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] [FLINK-34543][datastream]Support Full Partition Processing On Non-keyed DataStream [flink]

2024-03-12 Thread via GitHub
reswqa commented on code in PR #24398: URL: https://github.com/apache/flink/pull/24398#discussion_r1522571183 ## flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java: ## @@ -128,6 +128,24 @@ public class ExecutionOptions {

Re: [PR] [FLINK-34543][datastream]Support Full Partition Processing On Non-keyed DataStream [flink]

2024-03-12 Thread via GitHub
reswqa commented on code in PR #24398: URL: https://github.com/apache/flink/pull/24398#discussion_r1522567807 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/MapPartitionOperator.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]

2024-03-12 Thread via GitHub
reswqa commented on code in PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#discussion_r1522556968 ## flink-connector-opensearch/archunit-violations/4382f1f0-807a-45ff-97d8-42f72b6e9484: ## @@ -14,23 +14,5 @@ org.apache.flink.connector.opensearch.tabl

Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]

2024-03-12 Thread via GitHub
reswqa commented on code in PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#discussion_r152244 ## flink-connector-opensearch/archunit-violations/4382f1f0-807a-45ff-97d8-42f72b6e9484: ## @@ -14,23 +14,5 @@ org.apache.flink.connector.opensearch.tabl

Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]

2024-03-12 Thread via GitHub
PatrickRen commented on code in PR #721: URL: https://github.com/apache/flink-web/pull/721#discussion_r1522504285 ## docs/content/posts/2024-03-xx-release-1.19.0.md: ## @@ -0,0 +1,501 @@ +--- +authors: +- LincolnLee: + name: "Lincoln Lee" + twitter: lincoln_86xy + +date: "2024

[jira] [Updated] (FLINK-32953) [State TTL]resolve data correctness problem after ttl was changed

2024-03-12 Thread Jinzhong Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-32953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinzhong Li updated FLINK-32953: Fix Version/s: (was: 1.20.0) > [State TTL]resolve data correctness problem after ttl was chang

Re: [PR] [FLINK-34533][release] Add release note for version 1.19 [flink]

2024-03-12 Thread via GitHub
PatrickRen commented on code in PR #24394: URL: https://github.com/apache/flink/pull/24394#discussion_r1522493325 ## docs/content.zh/_index.md: ## @@ -85,7 +85,8 @@ under the License. For some reason Hugo will only allow linking to the release notes if there is a leading '/'

Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]

2024-03-12 Thread via GitHub
hajimeni commented on PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#issuecomment-1993462184 Hey, @schulzp. I have created a PR for testing. https://github.com/hajimeni/flink-connector-opensearch/pull/1 CI passes, so it still seems that the differe

[jira] [Updated] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-33798: Fix Version/s: 1.20.0 > Automatically clean up rocksdb logs when the task failover. >

[jira] [Commented] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825899#comment-17825899 ] Rui Fan commented on FLINK-33798: - Hi [~liming], IIUC, this JIRA is a file leak related

[jira] [Commented] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825898#comment-17825898 ] Rui Fan commented on FLINK-33798: - Merged to master(1.20) via: 398bb5068703a6461e2e52a90

[jira] [Updated] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-33798: Issue Type: Bug (was: Improvement) > Automatically clean up rocksdb logs when the task failover. > --

Re: [PR] [FLINK-33798][statebackend/rocksdb] automatically clean up rocksdb logs when the task exited. [flink]

2024-03-12 Thread via GitHub
1996fanrui merged PR #23922: URL: https://github.com/apache/flink/pull/23922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.ap

[jira] [Commented] (FLINK-34557) When the Flink task ends in application mode, there may be issues with the Znode and HDFS files not being deleted

2024-03-12 Thread tanliang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825895#comment-17825895 ] tanliang commented on FLINK-34557: -- My current temporary solution is to add a shutdownh

[jira] [Commented] (FLINK-34659) How to implement global sort in latest flink datastream API

2024-03-12 Thread Junyao Huang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825894#comment-17825894 ] Junyao Huang commented on FLINK-34659: -- [https://github.com/apache/flink/blob/94b55

[jira] [Updated] (FLINK-34659) How to implement global sort in latest flink datastream API

2024-03-12 Thread Junyao Huang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyao Huang updated FLINK-34659: - Attachment: image-2024-03-13-11-21-57-846.png > How to implement global sort in latest flink dat

[jira] [Updated] (FLINK-34659) How to implement global sort in latest flink datastream API

2024-03-12 Thread Junyao Huang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyao Huang updated FLINK-34659: - Description: [https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/datastream/datase

[jira] [Created] (FLINK-34659) How to implement global sort in latest flink datastream API

2024-03-12 Thread Junyao Huang (Jira)
Junyao Huang created FLINK-34659: Summary: How to implement global sort in latest flink datastream API Key: FLINK-34659 URL: https://issues.apache.org/jira/browse/FLINK-34659 Project: Flink

[jira] [Commented] (FLINK-34557) When the Flink task ends in application mode, there may be issues with the Znode and HDFS files not being deleted

2024-03-12 Thread tanliang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825893#comment-17825893 ] tanliang commented on FLINK-34557: -- hi [~mapohl] ,thanks for your comment, I'm sorry fo

[jira] [Comment Edited] (FLINK-34569) 'Streaming File Sink s3 end-to-end test' failed

2024-03-12 Thread Rob Young (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825887#comment-17825887 ] Rob Young edited comment on FLINK-34569 at 3/13/24 3:14 AM:

Re: [PR] [FLINK-34649][table] Migrate PushFilterIntoLegacyTableSourceScanRule … [flink]

2024-03-12 Thread via GitHub
damumu0625 commented on PR #24485: URL: https://github.com/apache/flink/pull/24485#issuecomment-1993272350 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] [FLINK-32076][checkpoint] Introduce file pool to reuse files [flink]

2024-03-12 Thread via GitHub
masteryhx commented on PR #24418: URL: https://github.com/apache/flink/pull/24418#issuecomment-1993258271 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]

2024-03-12 Thread via GitHub
lincoln-lil commented on PR #721: URL: https://github.com/apache/flink-web/pull/721#issuecomment-1993232847 @Myasuka Thanks! I've updated the pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[jira] [Comment Edited] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825669#comment-17825669 ] Rui Fan edited comment on FLINK-34655 at 3/13/24 2:46 AM: -- {quo

[jira] [Updated] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-34655: Fix Version/s: kubernetes-operator-1.8.0 (was: 1.8.0) > Autoscaler doesn't work for

[jira] [Commented] (FLINK-34569) 'Streaming File Sink s3 end-to-end test' failed

2024-03-12 Thread Rob Young (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825887#comment-17825887 ] Rob Young commented on FLINK-34569: --- Hi, I've had a look into it and I think the repea

Re: [PR] [FLINK-34618][table] Migrate SplitPythonConditionFromJoinRule to java. [flink]

2024-03-12 Thread via GitHub
liuyongvs commented on PR #24486: URL: https://github.com/apache/flink/pull/24486#issuecomment-1993138852 LGTM, only left a minor comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [FLINK-34618][table] Migrate SplitPythonConditionFromJoinRule to java. [flink]

2024-03-12 Thread via GitHub
liuyongvs commented on code in PR #24486: URL: https://github.com/apache/flink/pull/24486#discussion_r1522385972 ## flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/SplitPythonConditionFromJoinRule.java: ## @@ -0,0 +1,133 @@ +/* + *

Re: [PR] [FLINK-32076][checkpoint] Introduce file pool to reuse files [flink]

2024-03-12 Thread via GitHub
masteryhx commented on PR #24418: URL: https://github.com/apache/flink/pull/24418#issuecomment-1993089967 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]

2024-03-12 Thread via GitHub
Myasuka commented on code in PR #721: URL: https://github.com/apache/flink-web/pull/721#discussion_r1522360842 ## docs/content/posts/2024-03-xx-release-1.19.0.md: ## @@ -0,0 +1,501 @@ +--- +authors: +- LincolnLee: + name: "Lincoln Lee" + twitter: lincoln_86xy + +date: "2024-03

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row type [flink]

2024-03-12 Thread via GitHub
ukby1234 commented on PR #24029: URL: https://github.com/apache/flink/pull/24029#issuecomment-1992944538 @MartijnVisser can you help review this PR? and possibly https://github.com/apache/flink/pull/23881 as well? -- This is an automated message from the Apache Git Service. To respond to

[jira] [Updated] (FLINK-34647) Path normalization is allocation intensive

2024-03-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34647: --- Labels: pull-request-available (was: ) > Path normalization is allocation intensive > -

Re: [PR] [FLINK-34647][core] Optimize Path normalization [flink]

2024-03-12 Thread via GitHub
schlosna commented on PR #24473: URL: https://github.com/apache/flink/pull/24473#issuecomment-1992844641 Thanks for taking a look at this PR. > 1. What's your setup for the path ? We have checkpoints writing to a variety of file systems depending on the infrastructure, so it mi

[jira] [Assigned] (FLINK-25352) Update stability annotations to include the since and missedGraduations fields

2024-03-12 Thread Jing Ge (Jira)
[ https://issues.apache.org/jira/browse/FLINK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge reassigned FLINK-25352: --- Assignee: Jing Ge > Update stability annotations to include the since and missedGraduations fields

[jira] [Created] (FLINK-34658) Scala API unusable on Flink 1.18.1/Java 17 Docker image

2024-03-12 Thread Matthew Ernst (Jira)
Matthew Ernst created FLINK-34658: - Summary: Scala API unusable on Flink 1.18.1/Java 17 Docker image Key: FLINK-34658 URL: https://issues.apache.org/jira/browse/FLINK-34658 Project: Flink Iss

[jira] [Created] (FLINK-34657) Implement Lineage Graph for streaming API use cases

2024-03-12 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-34657: - Summary: Implement Lineage Graph for streaming API use cases Key: FLINK-34657 URL: https://issues.apache.org/jira/browse/FLINK-34657 Project: Flink Issue T

Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]

2024-03-12 Thread via GitHub
schulzp commented on PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#issuecomment-1992274982 Hey @hajimeni! > The CI error seems to be caused by differences in files under the archunit-violations path. > It was resolved in the following PR. https://githu

Re: [PR] Modify BarrierAlignmentUtilTest comments [flink]

2024-03-12 Thread via GitHub
HCTommy commented on PR #24476: URL: https://github.com/apache/flink/pull/24476#issuecomment-1992162943 @zhuzhurk Hi, I fixed a grammar error in the comments when testing the barrier. Could you please review in your available time. -- This is an automated message from the Apache Git Servi

Re: [PR] [hotfix] In case of unexpected errors do not loose the primary failur [flink]

2024-03-12 Thread via GitHub
rkhachatryan commented on code in PR #24487: URL: https://github.com/apache/flink/pull/24487#discussion_r1521803705 ## flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java: ## @@ -1046,27 +1046,32 @@ private void onTriggerFailure(

Re: [PR] [hotfix] In case of unexpected errors do not loose the primary failur [flink]

2024-03-12 Thread via GitHub
rkhachatryan commented on code in PR #24487: URL: https://github.com/apache/flink/pull/24487#discussion_r1521803705 ## flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java: ## @@ -1046,27 +1046,32 @@ private void onTriggerFailure(

Re: [PR] [hotfix] In case of unexpected errors do not loose the primary failur [flink]

2024-03-12 Thread via GitHub
flinkbot commented on PR #24487: URL: https://github.com/apache/flink/pull/24487#issuecomment-1992068365 ## CI report: * d07a37b06fc847c1f2c6ce148a918c2490f2490c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[PR] [hotfix] In case of unexpected errors do not loose the primary failur [flink]

2024-03-12 Thread via GitHub
pnowojski opened a new pull request, #24487: URL: https://github.com/apache/flink/pull/24487 Unexpected error can be for example NPE ## Verifying this change This ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825698#comment-17825698 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 3:45 PM: --

Re: [PR] [FLINK-34533][release] Add release note for version 1.19 [flink]

2024-03-12 Thread via GitHub
lincoln-lil commented on code in PR #24394: URL: https://github.com/apache/flink/pull/24394#discussion_r1521683726 ## docs/content.zh/release-notes/flink-1.19.md: ## @@ -0,0 +1,362 @@ +--- +title: "Release Notes - Flink 1.19" +--- + + +# Release notes - Flink 1.19 + +These relea

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825698#comment-17825698 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 3:20 PM: --

[jira] [Commented] (FLINK-23542) Upgrade Checkstyle to at least 8.29

2024-03-12 Thread Ryan Skraba (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825712#comment-17825712 ] Ryan Skraba commented on FLINK-23542: - This has become slightly more important : the

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825698#comment-17825698 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 3:12 PM: --

Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]

2024-03-12 Thread via GitHub
lincoln-lil commented on code in PR #721: URL: https://github.com/apache/flink-web/pull/721#discussion_r1521623410 ## docs/content/posts/2024-03-xx-release-1.19.0.md: ## @@ -0,0 +1,456 @@ +--- Review Comment: @MartijnVisser As we discussed at the sync meeting, I've added the

Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]

2024-03-12 Thread via GitHub
lincoln-lil commented on PR #721: URL: https://github.com/apache/flink-web/pull/721#issuecomment-1991840142 @xintongsong @rmoff As per the conclusion of our discussion, I've tweaked these examples and tried to keep them to just describing what the functionality looks like (and avoiding over

[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825698#comment-17825698 ] Matthias Pohl commented on FLINK-34227: --- [~chesnay] supported the investigation. H

[jira] [Created] (FLINK-34656) Generated code for `ITEM` operator should return null when getting element of a null map/array/row

2024-03-12 Thread yisha zhou (Jira)
yisha zhou created FLINK-34656: -- Summary: Generated code for `ITEM` operator should return null when getting element of a null map/array/row Key: FLINK-34656 URL: https://issues.apache.org/jira/browse/FLINK-34656

Re: [PR] [FLINK-34334][state] Add sub-task level RocksDB file count metrics [flink]

2024-03-12 Thread via GitHub
hejufang commented on code in PR #24322: URL: https://github.com/apache/flink/pull/24322#discussion_r1521505824 ## flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBProperty.java: ## @@ -31,55 +31,92 @@ */ @Internal

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825654#comment-17825654 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 1:09 PM: --

Re: [PR] [FLINK-34152][autoscaler] Refactor the MemoryTuning to avoid the huge method [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
1996fanrui commented on code in PR #795: URL: https://github.com/apache/flink-kubernetes-operator/pull/795#discussion_r1521431571 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java: ## @@ -92,24 +99,51 @@ public static ConfigChanges tuneTaskM

[jira] [Commented] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Rui Fan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825669#comment-17825669 ] Rui Fan commented on FLINK-34655: - {quote}But the vertex parallelism overrides feature w

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825654#comment-17825654 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 1:00 PM: --

Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
gyfora commented on PR #797: URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991544918 > Thanks @gyfora for the valuable feedback! > > > Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see: https://github

Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
1996fanrui commented on PR #797: URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991537856 Thanks @gyfora for the valuable feedback! > Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see: https://github.

Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
gyfora commented on PR #797: URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991513679 But still it will not make it work fully for 1.15 as the aggregated metrics need to be back ported on the Flink side (similar to other features required for the autoscaler

Re: [PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
gyfora commented on PR #797: URL: https://github.com/apache/flink-kubernetes-operator/pull/797#issuecomment-1991512469 Instead of a new rest client you could also do what is done in the Kubernetes-operator module, see: https://github.com/apache/flink-kubernetes-operator/blob/main/flink-

[jira] [Comment Edited] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825657#comment-17825657 ] Gyula Fora edited comment on FLINK-34655 at 3/12/24 12:12 PM:

[jira] [Commented] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825657#comment-17825657 ] Gyula Fora commented on FLINK-34655: Also this issue is fixed in the Kubernetes-oper

[jira] [Updated] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-34655: --- Priority: Major (was: Blocker) > Autoscaler doesn't work for flink 1.15 > -

[jira] [Commented] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825656#comment-17825656 ] Gyula Fora commented on FLINK-34655: But the vertex parallelism overrides feature wa

[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825654#comment-17825654 ] Matthias Pohl commented on FLINK-34227: --- The [findings of my initial analysis|htt

[jira] [Commented] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825655#comment-17825655 ] Gyula Fora commented on FLINK-34655: The bigger issue is that aggregated busy time m

Re: [PR] feat: autoscaling decision parallelism improvement [FLINK-34563] [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
mxm commented on PR #787: URL: https://github.com/apache/flink-kubernetes-operator/pull/787#issuecomment-1991480345 I think there are some really good ideas in this PR which we might consider in the future, but for the sake of simplicity it might be better to leave things as-is for now. Th

Re: [PR] [FLINK-34152][autoscaler] Refactor the MemoryTuning to avoid the huge method [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
mxm commented on code in PR #795: URL: https://github.com/apache/flink-kubernetes-operator/pull/795#discussion_r1521324931 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/tuning/MemoryTuning.java: ## @@ -92,24 +99,51 @@ public static ConfigChanges tuneTaskManagerM

Re: [PR] [FLINK-34543][datastream]Support Full Partition Processing On Non-keyed DataStream [flink]

2024-03-12 Thread via GitHub
WencongLiu commented on code in PR #24398: URL: https://github.com/apache/flink/pull/24398#discussion_r1521322872 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/MapPartitionOperator.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
LadyForest commented on PR #24390: URL: https://github.com/apache/flink/pull/24390#issuecomment-1991461563 Hi [XComp](https://github.com/XComp), thanks so much for the time and effort you've put into the review and discussion :) -- This is an automated message from the Apache Git Service.

Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]

2024-03-12 Thread via GitHub
hajimeni commented on PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#issuecomment-1991441614 Hey @schulzp. Thank you for creating this PR. It would be very helpful if this PR gets merged. The CI error seems to be caused by differences in files under the `a

[jira] [Comment Edited] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825639#comment-17825639 ] Matthias Pohl edited comment on FLINK-34227 at 3/12/24 11:24 AM: -

[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager

2024-03-12 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825639#comment-17825639 ] Matthias Pohl commented on FLINK-34227: --- My suspicion is that there's a bug in the

Re: [PR] [FLINK-30481][FLIP-277] GlueCatalog Implementation [flink-connector-aws]

2024-03-12 Thread via GitHub
wckdman commented on PR #47: URL: https://github.com/apache/flink-connector-aws/pull/47#issuecomment-1991405676 Any news here? :) I'm really looking forward to see this feature 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[jira] [Updated] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34655: --- Labels: pull-request-available (was: ) > Autoscaler doesn't work for flink 1.15 > -

[PR] [FLINK-34655] Autoscaler doesn't work for flink 1.15 [flink-kubernetes-operator]

2024-03-12 Thread via GitHub
1996fanrui opened a new pull request, #797: URL: https://github.com/apache/flink-kubernetes-operator/pull/797 flink-ubernetes-operator is committed to supporting the latest 4 flink minor versions, and autoscaler is a part of flink-ubernetes-operator. Currently, the latest 4 flink minor vers

Re: [PR] [FLINK-34615]Split `ExternalizedCheckpointCleanup` out of `Checkpoint… [flink]

2024-03-12 Thread via GitHub
Zakelly commented on code in PR #24461: URL: https://github.com/apache/flink/pull/24461#discussion_r1521260155 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java: ## @@ -520,12 +521,40 @@ public void setTolerableCheckpointFailu

[jira] [Created] (FLINK-34655) Autoscaler doesn't work for flink 1.15

2024-03-12 Thread Rui Fan (Jira)
Rui Fan created FLINK-34655: --- Summary: Autoscaler doesn't work for flink 1.15 Key: FLINK-34655 URL: https://issues.apache.org/jira/browse/FLINK-34655 Project: Flink Issue Type: Bug Compon

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
XComp commented on code in PR #24390: URL: https://github.com/apache/flink/pull/24390#discussion_r1521248255 ## flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/table/FileSystemOutputFormatTest.java: ## @@ -282,7 +278,7 @@ private FileSystemOu

Re: [PR] [FLINK-32076][checkpoint] Introduce file pool to reuse files [flink]

2024-03-12 Thread via GitHub
masteryhx commented on PR #24418: URL: https://github.com/apache/flink/pull/24418#issuecomment-1991306532 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [FLINK-34654] Add Special Thanks Page on the Flink Website [flink-web]

2024-03-12 Thread via GitHub
wuchong commented on PR #725: URL: https://github.com/apache/flink-web/pull/725#issuecomment-1991281258 @dannycranmer @hlteoh37 could you help to review the AWS part? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] [FLINK-34654] Add Special Thanks Page on the Flink Website [flink-web]

2024-03-12 Thread via GitHub
wuchong commented on code in PR #725: URL: https://github.com/apache/flink-web/pull/725#discussion_r1521200340 ## docs/content/what-is-flink/special-thanks.md: ## @@ -0,0 +1,48 @@ +--- +title: Special Thanks +bookCollapseSection: false +weight: 9 +--- + + +# Special Thanks + +##

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
LadyForest commented on code in PR #24390: URL: https://github.com/apache/flink/pull/24390#discussion_r1521195360 ## flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/table/FileSystemOutputFormatTest.java: ## @@ -37,24 +39,44 @@ import java.io

Re: [PR] [FLINK-34654] Add Special Thanks Page on the Flink Website [flink-web]

2024-03-12 Thread via GitHub
Zakelly commented on code in PR #725: URL: https://github.com/apache/flink-web/pull/725#discussion_r1521191193 ## docs/content/what-is-flink/special-thanks.md: ## @@ -0,0 +1,48 @@ +--- +title: Special Thanks +bookCollapseSection: false +weight: 9 +--- + + +# Special Thanks + +##

Re: [PR] [FLINK-34654] Add Special Thanks Page on the Flink Website [flink-web]

2024-03-12 Thread via GitHub
rmetzger commented on code in PR #725: URL: https://github.com/apache/flink-web/pull/725#discussion_r1521186823 ## docs/content/what-is-flink/special-thanks.md: ## @@ -0,0 +1,48 @@ +--- +title: Special Thanks +bookCollapseSection: false +weight: 9 +--- + + +# Special Thanks + +#

Re: [PR] [FLINK-34618][table] Migrate SplitPythonConditionFromJoinRule to java. [flink]

2024-03-12 Thread via GitHub
flinkbot commented on PR #24486: URL: https://github.com/apache/flink/pull/24486#issuecomment-1991230364 ## CI report: * 83e67a1d6988fe08173cdbcf161f6097dde2fecb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[jira] [Updated] (FLINK-34618) Migrate SplitPythonConditionFromJoinRule

2024-03-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34618: --- Labels: pull-request-available (was: ) > Migrate SplitPythonConditionFromJoinRule > ---

[PR] [FLINK-34618][table] Migrate SplitPythonConditionFromJoinRule to java. [flink]

2024-03-12 Thread via GitHub
damumu0625 opened a new pull request, #24486: URL: https://github.com/apache/flink/pull/24486 ## What is the purpose of the change The PR migrates SplitPythonConditionFromJoinRule to java it doesn't touch SplitPythonConditionFromJoinRuleTest to be sure that java version con

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
LadyForest commented on code in PR #24390: URL: https://github.com/apache/flink/pull/24390#discussion_r1521155291 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemOutputFormat.java: ## @@ -96,6 +99,22 @@ private FileSystemOu

[jira] [Updated] (FLINK-34654) Add "Special Thanks" Page on the Flink Website

2024-03-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34654: --- Labels: pull-request-available (was: ) > Add "Special Thanks" Page on the Flink Website > -

Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]

2024-03-12 Thread via GitHub
LadyForest commented on code in PR #24390: URL: https://github.com/apache/flink/pull/24390#discussion_r1521155291 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemOutputFormat.java: ## @@ -96,6 +99,22 @@ private FileSystemOu

Re: [PR] [FLINK-32076][checkpoint] Introduce file pool to reuse files [flink]

2024-03-12 Thread via GitHub
masteryhx commented on code in PR #24418: URL: https://github.com/apache/flink/pull/24418#discussion_r1521151391 ## flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/BlockingPhysicalFilePool.java: ## @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Softw

  1   2   >