Re: [PR] [FLINK-33892][runtime] Support Job Recovery from JobMaster Failures for Batch Jobs. [flink]

2024-05-22 Thread via GitHub
JunRuiLee commented on code in PR #24771: URL: https://github.com/apache/flink/pull/24771#discussion_r1611044994 ## flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java: ## @@ -0,0 +1,840 @@ +/* + * Licensed to the

Re: [PR] [FLINK-35359][config] General Improvement to Configuration for Flink 2.0 [flink]

2024-05-22 Thread via GitHub
Sxnan commented on PR #24815: URL: https://github.com/apache/flink/pull/24815#issuecomment-2126293155 @xintongsong Thanks for the review! I updated the PR accordingly. Please take another look. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [FLINK-34123][core][type] Introduce built-in serialization support for map and lists [flink]

2024-05-22 Thread via GitHub
X-czh commented on PR #24634: URL: https://github.com/apache/flink/pull/24634#issuecomment-2126292943 > Thanks @X-czh, I think we only need two commits. One for the implementation and one for documentation. Thanks for the notice, @reswqa. I've rearranged the commits to have only two

Re: [PR] [FLINK-35049][state] Implement Map Async State API for ForStStateBackend [flink]

2024-05-22 Thread via GitHub
jectpro7 commented on code in PR #24812: URL: https://github.com/apache/flink/pull/24812#discussion_r1610081555 ## flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStDBBunchPutRequest.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the

Re: [PR] [FLINK-34123][core][type] Introduce built-in serialization support for map and lists [flink]

2024-05-22 Thread via GitHub
reswqa commented on PR #24634: URL: https://github.com/apache/flink/pull/24634#issuecomment-2126264511 Thanks @X-czh. Let's wait for the CI to pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [FLINK-34123][core][type] Introduce built-in serialization support for map and lists [flink]

2024-05-22 Thread via GitHub
reswqa commented on code in PR #24634: URL: https://github.com/apache/flink/pull/24634#discussion_r1610998637 ## docs/content/docs/dev/datastream/fault-tolerance/serialization/types_serialization.md: ## @@ -37,15 +37,16 @@ Flink places some restrictions on the type of elements

Re: [PR] [FLINK-35240][Connectors][format]Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record for csv format [flink]

2024-05-22 Thread via GitHub
GOODBOY008 commented on code in PR #24730: URL: https://github.com/apache/flink/pull/24730#discussion_r1610997416 ## flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java: ## @@ -98,16 +108,17 @@ static CsvBulkWriter forPojo(Class pojoClass,

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610987543 ## flink-connector-aws-base/pom.xml: ## @@ -76,6 +76,12 @@ under the License. test + Review Comment: removed

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610987837 ## flink-connector-aws/flink-connector-sqs/src/main/java/org.apache.flink.connector.sqs/sink/SqsSinkWriter.java: ## @@ -0,0 +1,272 @@ +/* + *

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610983272 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/testutils/SqsTestUtils.java: ## @@ -0,0 +1,76 @@ +/* + *

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610974483 ## flink-connector-aws-base/src/test/java/org/apache/flink/connector/aws/testutils/AWSServicesTestUtils.java: ## @@ -114,6 +116,14 @@ public static

[jira] [Assigned] (FLINK-35411) Optimize wait logic in draining of async state requests

2024-05-22 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zakelly Lan reassigned FLINK-35411: --- Assignee: Yanfei Lei (was: Zakelly Lan) > Optimize wait logic in draining of async state

[jira] [Assigned] (FLINK-35412) Batch execution of async state request callback

2024-05-22 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zakelly Lan reassigned FLINK-35412: --- Assignee: Zakelly Lan > Batch execution of async state request callback >

[jira] [Commented] (FLINK-35411) Optimize wait logic in draining of async state requests

2024-05-22 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848812#comment-17848812 ] Zakelly Lan commented on FLINK-35411: - [~spoon-lz] Actually this is found by some our internal

[jira] [Assigned] (FLINK-35411) Optimize wait logic in draining of async state requests

2024-05-22 Thread Zakelly Lan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zakelly Lan reassigned FLINK-35411: --- Assignee: Zakelly Lan > Optimize wait logic in draining of async state requests >

Re: [PR] [FLINK-35355][State] Internal async aggregating state and corresponding state descriptor [flink]

2024-05-22 Thread via GitHub
Zakelly commented on PR #24810: URL: https://github.com/apache/flink/pull/24810#issuecomment-2126218813 @jectpro7 Seems the author of this commit is different from the committer? Do you mind change that? Or I can merge this as it is. -- This is an automated message from the Apache Git

Re: [PR] [FLINK-34123][core][type] Introduce built-in serialization support for map and lists [flink]

2024-05-22 Thread via GitHub
X-czh commented on PR #24634: URL: https://github.com/apache/flink/pull/24634#issuecomment-2126217002 @reswqa Thanks for the review. I've rebased on latest master branch, and it has been highlighted in the doc content as follows: > Currently, only `Map`, `List` and its super interface

[jira] [Assigned] (FLINK-34914) FLIP-436: Introduce Catalog-related Syntax

2024-05-22 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-34914: -- Assignee: Yubin Li > FLIP-436: Introduce Catalog-related Syntax >

Re: [PR] [FLINK-35047][state] Shutdown StateExecutors when ForStKeyedStateBackend is closed [flink]

2024-05-22 Thread via GitHub
masteryhx closed pull request #24768: [FLINK-35047][state] Shutdown StateExecutors when ForStKeyedStateBackend is closed URL: https://github.com/apache/flink/pull/24768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610935419 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/testutils/SqsTestUtils.java: ## @@ -0,0 +1,76 @@ +/* + *

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610935001 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/SqsExceptionClassifiersTest.java: ## @@ -0,0 +1,72 @@

[jira] [Assigned] (FLINK-34123) Introduce built-in serialization support for Map and List

2024-05-22 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-34123: -- Assignee: Zhanghao Chen > Introduce built-in serialization support for Map and List >

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610929881 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/SqsExceptionClassifiersTest.java: ## @@ -0,0 +1,72 @@

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610929610 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/SqsExceptionClassifiersTest.java: ## @@ -0,0 +1,72 @@

Re: [PR] [FLINK-34123][core][type] Introduce built-in serialization support for map and lists [flink]

2024-05-22 Thread via GitHub
reswqa commented on code in PR #24634: URL: https://github.com/apache/flink/pull/24634#discussion_r1610929345 ## docs/content/docs/dev/datastream/fault-tolerance/serialization/types_serialization.md: ## @@ -37,15 +37,16 @@ Flink places some restrictions on the type of elements

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610928629 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/SqsSinkWriterTest.java: ## @@ -0,0 +1,105 @@ +/* + *

[jira] [Created] (FLINK-35427) Support fail-on-unknown-field config in json format

2024-05-22 Thread huhuan (Jira)
huhuan created FLINK-35427: -- Summary: Support fail-on-unknown-field config in json format Key: FLINK-35427 URL: https://issues.apache.org/jira/browse/FLINK-35427 Project: Flink Issue Type:

Re: [PR] [FLINK-35240][Connectors][format]Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record for csv format [flink]

2024-05-22 Thread via GitHub
robobario commented on code in PR #24730: URL: https://github.com/apache/flink/pull/24730#discussion_r1610925700 ## flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java: ## @@ -98,16 +108,17 @@ static CsvBulkWriter forPojo(Class pojoClass,

Re: [PR] [FLINK-35426][table-planner] Change the distribution of DynamicFilteringDataCollector to Broadcast [flink]

2024-05-22 Thread via GitHub
flinkbot commented on PR #24830: URL: https://github.com/apache/flink/pull/24830#issuecomment-2126172102 ## CI report: * fdeb2381c152dc3339779d645717ff9099664011 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[jira] [Updated] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast

2024-05-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35426: --- Labels: pull-request-available (was: ) > Change the distribution of

[PR] [FLINK-35426][table-planner] Change the distribution of DynamicFilteringDataCollector to Broadcast [flink]

2024-05-22 Thread via GitHub
SinBex opened a new pull request, #24830: URL: https://github.com/apache/flink/pull/24830 ## What is the purpose of the change Currently, the DynamicFilteringDataCollector is utilized in the dynamic partition pruning feature of batch jobs to collect the partition information

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610910711 ## flink-connector-aws/flink-connector-sqs/src/main/java/org.apache.flink.connector.sqs/sink/SqsStateSerializer.java: ## @@ -0,0 +1,54 @@ +/* + *

[jira] [Updated] (FLINK-35421) Schema Operator blocking forever when Akka Rpc timeout

2024-05-22 Thread hk__lrzy (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hk__lrzy updated FLINK-35421: - Description: When SchemaOperator restart and there have no checkpoint before, SchemaOperator will not

[jira] [Commented] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast

2024-05-22 Thread Zhu Zhu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848801#comment-17848801 ] Zhu Zhu commented on FLINK-35426: - Good point! [~xiasun] The task is assigned to you. Feel free to open

[jira] [Assigned] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast

2024-05-22 Thread Zhu Zhu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu reassigned FLINK-35426: --- Assignee: xingbe > Change the distribution of DynamicFilteringDataCollector to Broadcast >

Re: [PR] [FLINK-31664] Implement ARRAY_INTERSECT function [flink]

2024-05-22 Thread via GitHub
liuyongvs commented on PR #24526: URL: https://github.com/apache/flink/pull/24526#issuecomment-2126150641 fix conflicts, @twalthr @dawidwys @snuyanzin and will you help review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [FLINK-31664] Implement ARRAY_INTERSECT function [flink]

2024-05-22 Thread via GitHub
liuyongvs commented on PR #24526: URL: https://github.com/apache/flink/pull/24526#issuecomment-2126150525 Conclusion: Since there are no objections, then we will support it with deduplication semantics. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast

2024-05-22 Thread xingbe (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848800#comment-17848800 ] xingbe commented on FLINK-35426: [~zhuzh] Could you please assign this ticket to me? Thanks. > Change

[jira] [Created] (FLINK-35426) Change the distribution of DynamicFilteringDataCollector to Broadcast

2024-05-22 Thread xingbe (Jira)
xingbe created FLINK-35426: -- Summary: Change the distribution of DynamicFilteringDataCollector to Broadcast Key: FLINK-35426 URL: https://issues.apache.org/jira/browse/FLINK-35426 Project: Flink

Re: [PR] [FLINK-35351][checkpoint] Fix fail during restore from unaligned chec… [flink]

2024-05-22 Thread via GitHub
ldadima commented on code in PR #24784: URL: https://github.com/apache/flink/pull/24784#discussion_r1609545011 ## flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java: ## @@ -421,7 +423,27 @@ public void

Re: [PR] [FLINK-35305]Amazon SQS Sink Connector [flink-connector-aws]

2024-05-22 Thread via GitHub
19priyadhingra commented on code in PR #141: URL: https://github.com/apache/flink-connector-aws/pull/141#discussion_r1610896956 ## flink-connector-aws/flink-connector-sqs/src/test/java/org.apache.flink/connector.sqs/sink/SqsSinkITCase.java: ## @@ -0,0 +1,121 @@ +/* + *

Re: [PR] [FLINK-35355][State] Internal async aggregating state and corresponding state descriptor [flink]

2024-05-22 Thread via GitHub
jectpro7 commented on PR #24810: URL: https://github.com/apache/flink/pull/24810#issuecomment-2126086682 Hi @Zakelly, I've addressed comments, please help double review. Thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[jira] [Created] (FLINK-35425) Support convert Freshness to cron expression for full refresh mode

2024-05-22 Thread dalongliu (Jira)
dalongliu created FLINK-35425: - Summary: Support convert Freshness to cron expression for full refresh mode Key: FLINK-35425 URL: https://issues.apache.org/jira/browse/FLINK-35425 Project: Flink

[jira] [Commented] (FLINK-35421) Schema Operator blocking forever when Akka Rpc timeout

2024-05-22 Thread hk__lrzy (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848793#comment-17848793 ] hk__lrzy commented on FLINK-35421: -- [~kunni]  Thank your for relay,  i already have a  pull request to

[jira] [Updated] (FLINK-35421) Schema Operator blocking forever when Akka Rpc timeout

2024-05-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35421: --- Labels: pull-request-available (was: ) > Schema Operator blocking forever when Akka Rpc

[PR] [FLINK-35421]fixed schema operator blocking when restart [flink-cdc]

2024-05-22 Thread via GitHub
hk-lrzy opened a new pull request, #3350: URL: https://github.com/apache/flink-cdc/pull/3350 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]

2024-05-22 Thread via GitHub
superdiaodiao commented on code in PR #24773: URL: https://github.com/apache/flink/pull/24773#discussion_r1610884892 ## flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java: ## @@ -0,0 +1,52 @@ +/* + * Licensed to

[jira] [Comment Edited] (FLINK-31223) sql-client.sh fails to start with ssl enabled

2024-05-22 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840864#comment-17840864 ] Weijie Guo edited comment on FLINK-31223 at 5/23/24 2:17 AM: - master(1.20)

[jira] [Updated] (FLINK-31223) sql-client.sh fails to start with ssl enabled

2024-05-22 Thread Weijie Guo (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-31223: --- Fix Version/s: 1.19.1 > sql-client.sh fails to start with ssl enabled >

Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to g… [flink]

2024-05-22 Thread via GitHub
reswqa merged PR #24742: URL: https://github.com/apache/flink/pull/24742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to g… [flink]

2024-05-22 Thread via GitHub
reswqa commented on PR #24742: URL: https://github.com/apache/flink/pull/24742#issuecomment-2126065522 Thanks for the backport. Could you squash these two commits, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to [flink]

2024-05-22 Thread via GitHub
reswqa commented on code in PR #24741: URL: https://github.com/apache/flink/pull/24741#discussion_r1610874737 ## flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/SqlClientTestBase.java: ## @@ -0,0 +1,119 @@ +/* + * Licensed to the Apache Software

Re: [PR] [FLINK-35129][cdc][postgres] Add checkpoint cycle option for commit offsets [flink-cdc]

2024-05-22 Thread via GitHub
loserwang1024 commented on code in PR #3349: URL: https://github.com/apache/flink-cdc/pull/3349#discussion_r1610861124 ## flink-cdc-connect/flink-cdc-source-connectors/flink-connector-postgres-cdc/src/main/java/org/apache/flink/cdc/connectors/postgres/source/PostgresDialect.java:

Re: [PR] [hotfix][rest] Add getter methods for SubtasksTimesInfo to get all values easily [flink]

2024-05-22 Thread via GitHub
1996fanrui merged PR #24825: URL: https://github.com/apache/flink/pull/24825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [hotfix][rest] Add getter methods for SubtasksTimesInfo to get all values easily [flink]

2024-05-22 Thread via GitHub
1996fanrui commented on PR #24825: URL: https://github.com/apache/flink/pull/24825#issuecomment-2126057162 Thanks @RocMarshal for the quick review, merging~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [hotfix][rest] Add getter methods for SubtasksTimesInfo to get all values easily [flink]

2024-05-22 Thread via GitHub
1996fanrui merged PR #24824: URL: https://github.com/apache/flink/pull/24824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (FLINK-35421) Schema Operator blocking forever when Akka Rpc timeout

2024-05-22 Thread LvYanquan (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848788#comment-17848788 ] LvYanquan commented on FLINK-35421: --- Thanks for pointing out this issue. I am willing to fix it. >

[jira] [Updated] (FLINK-35417) JobManager and TaskManager support merging and run in a single process

2024-05-22 Thread Xintong Song (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-35417: - Component/s: Runtime / Coordination (was: API / Core) > JobManager and

[jira] [Updated] (FLINK-35129) Postgres source commits the offset after every multiple checkpoint cycles.

2024-05-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35129: --- Labels: pull-request-available (was: ) > Postgres source commits the offset after every

[PR] [FLINK-35129][cdc][postgres] Add checkpoint cycle option for commit offsets [flink-cdc]

2024-05-22 Thread via GitHub
morazow opened a new pull request, #3349: URL: https://github.com/apache/flink-cdc/pull/3349 https://issues.apache.org/jira/browse/FLINK-35129 - Adds option for checkpoint cycle parameter - Commits PG offsets on every multiple of checkpoint cycle - Updates docs with new

[jira] [Updated] (FLINK-35424) Elasticsearch connector 8 supports SSL context

2024-05-22 Thread Mingliang Liu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated FLINK-35424: -- Description: In  FLINK-34369, we added SSL support for the base Elasticsearch sink class

[jira] [Updated] (FLINK-35424) Elasticsearch connector 8 supports SSL context

2024-05-22 Thread Mingliang Liu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated FLINK-35424: -- Description: In  FLINK-34369, we added SSL support for the base Elasticsearch sink class that

[jira] [Updated] (FLINK-35424) Elasticsearch connector 8 supports SSL context

2024-05-22 Thread Mingliang Liu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated FLINK-35424: -- Description: In  (was: The current Flink ElasticSearch connector does not support SSL

[jira] [Created] (FLINK-35424) Elasticsearch connector 8 supports SSL context

2024-05-22 Thread Mingliang Liu (Jira)
Mingliang Liu created FLINK-35424: - Summary: Elasticsearch connector 8 supports SSL context Key: FLINK-35424 URL: https://issues.apache.org/jira/browse/FLINK-35424 Project: Flink Issue Type:

Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]

2024-05-22 Thread via GitHub
snuyanzin commented on code in PR #24773: URL: https://github.com/apache/flink/pull/24773#discussion_r1610775137 ## flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java: ## @@ -0,0 +1,52 @@ +/* + * Licensed to the

Re: [PR] remove duplicated word [flink]

2024-05-22 Thread via GitHub
flinkbot commented on PR #24829: URL: https://github.com/apache/flink/pull/24829#issuecomment-212581 ## CI report: * b0bd48bd8861803713456e8b8e3ca13be7d01ec1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[PR] remove duplicated word [flink]

2024-05-22 Thread via GitHub
naferx opened a new pull request, #24829: URL: https://github.com/apache/flink/pull/24829 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on

Re: [PR] [FLINK-35423][table] ARRAY_EXCEPT should follow set semantics [flink]

2024-05-22 Thread via GitHub
flinkbot commented on PR #24828: URL: https://github.com/apache/flink/pull/24828#issuecomment-2125833919 ## CI report: * cebfd0b8fa496b5cc97280c044352a71dbd75b1e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[jira] [Assigned] (FLINK-35423) ARRAY_EXCEPT should support set semantics

2024-05-22 Thread Sergey Nuyanzin (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin reassigned FLINK-35423: --- Assignee: Sergey Nuyanzin > ARRAY_EXCEPT should support set semantics >

[jira] [Updated] (FLINK-35423) ARRAY_EXCEPT should support set semantics

2024-05-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35423: --- Labels: pull-request-available (was: ) > ARRAY_EXCEPT should support set semantics >

[PR] [FLINK-35423][table] ARRAY_EXCEPT should follow set semantics [flink]

2024-05-22 Thread via GitHub
snuyanzin opened a new pull request, #24828: URL: https://github.com/apache/flink/pull/24828 ## What is the purpose of the change The idea of the PR is making of `ARRAY_EXCEPT` following set semantics ## Brief change log

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
ViktorCosenza commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2125705836 I see, Ive got the impression that they were forgotten, not purposely left out because no tests covered writing nested structures ( if there were, the tests would fail and the

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
JingGe commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2125686671 It looks like those methods were skipped on purpose in #17542 and https://github.com/apache/flink/pull/17542#issuecomment-1954552466 -- This is an automated message from the Apache Git

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
ViktorCosenza commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2125670540 > > Do you have any hints about the background info? > > Not really, I found out this issue because we were trying to save Parquet files to S3 and the writer would fail due

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
JingGe commented on code in PR #24795: URL: https://github.com/apache/flink/pull/24795#discussion_r1610586619 ## flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java: ## @@ -381,9 +381,16 @@ private MapWriter(LogicalType

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
ViktorCosenza commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2125663861 > Do you have any hints about the background info? Not really, I found out this issue because we were trying to save Parquet files to S3 and the writer would fail due to

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
JingGe commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2125655594 Thanks @ViktorCosenza for taking care of it. The PR looks overall good. Not sure why those `write(ArrayData arrayData, int ordinal)` methods didn't get implemented. It should be trivial

Re: [PR] chore: Update pekko version [flink]

2024-05-22 Thread via GitHub
He-Pin commented on PR #24823: URL: https://github.com/apache/flink/pull/24823#issuecomment-2125584766 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [FLINK-35240][Connectors][format]Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record for csv format [flink]

2024-05-22 Thread via GitHub
afedulov commented on code in PR #24730: URL: https://github.com/apache/flink/pull/24730#discussion_r1610487523 ## flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java: ## @@ -98,16 +108,17 @@ static CsvBulkWriter forPojo(Class pojoClass,

Re: [PR] [FLINK-20400][e2e] Migrate test_streaming_sql.sh [flink]

2024-05-22 Thread via GitHub
jeyhunkarimov commented on code in PR #24776: URL: https://github.com/apache/flink/pull/24776#discussion_r1610462318 ## flink-end-to-end-tests/flink-stream-sql-test/src/test/java/org/apache/flink/sql/tests/StreamSQLTestProgramScalaPlannerITCase.java: ## @@ -0,0 +1,90 @@ +/* + *

[jira] [Commented] (FLINK-35417) JobManager and TaskManager support merging and run in a single process

2024-05-22 Thread Jing Ge (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848706#comment-17848706 ] Jing Ge commented on FLINK-35417: - [~jeyhunkarimov] assigned you, please pay attention to comments

Re: [PR] [FLINK-35216] Support for RETURNING clause of JSON_QUERY [flink]

2024-05-22 Thread via GitHub
snuyanzin commented on PR #24704: URL: https://github.com/apache/flink/pull/24704#issuecomment-2125483868 hm... I tested with the latest version and noticed this strange behavior: I use same query ```sql SELECT JSON_QUERY('{"a":[{"c":null},{"c":"c2"}]}', 'lax $.a[*].c' RETURNING

[jira] [Assigned] (FLINK-35417) JobManager and TaskManager support merging and run in a single process

2024-05-22 Thread Jing Ge (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge reassigned FLINK-35417: --- Assignee: Jeyhun Karimov > JobManager and TaskManager support merging and run in a single process

Re: [PR] [FLINK-35216] Support for RETURNING clause of JSON_QUERY [flink]

2024-05-22 Thread via GitHub
dawidwys commented on code in PR #24704: URL: https://github.com/apache/flink/pull/24704#discussion_r1610404628 ## flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/JsonQueryOnErrorEmptyArgumentTypeStrategy.java: ## @@ -0,0 +1,57 @@

Re: [PR] [FLINK-33212][runtime] add job status changed listener for lineage [flink]

2024-05-22 Thread via GitHub
HuangZhenQiu commented on PR #24754: URL: https://github.com/apache/flink/pull/24754#issuecomment-2125397607 @davidradl The throwable in executors are caught already in Execution environment. If there is a better idea to provide extra info for customers, I am glad to adopt. -- This

Re: [PR] [FLINK-31223][sqlgateway] Introduce getFlinkConfigurationOptions to [flink]

2024-05-22 Thread via GitHub
davidradl commented on PR #24741: URL: https://github.com/apache/flink/pull/24741#issuecomment-2125234532 @reswqa the CI output shows that the config.yaml is not picked up. This was moved into the base test calls by the fix. On the face of it it looks like the @BeforeEach is not being

Re: [PR] [FLINK-35216] Support for RETURNING clause of JSON_QUERY [flink]

2024-05-22 Thread via GitHub
snuyanzin commented on code in PR #24704: URL: https://github.com/apache/flink/pull/24704#discussion_r1610255280 ## flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/JsonQueryOnErrorEmptyArgumentTypeStrategy.java: ## @@ -0,0 +1,57 @@

[jira] [Commented] (FLINK-35417) JobManager and TaskManager support merging and run in a single process

2024-05-22 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848676#comment-17848676 ] melin commented on FLINK-35417: --- The current deployment model has encountered challenges in many projects.

[jira] [Commented] (FLINK-34935) TIMESTAMP_LTZ type Unsupported when using JdbcCatalog to read from Postgres

2024-05-22 Thread Pietro (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848664#comment-17848664 ] Pietro commented on FLINK-34935: Related issue: FLINK-35053 > TIMESTAMP_LTZ type Unsupported when using

[jira] [Updated] (FLINK-35423) ARRAY_EXCEPT should support set semantics

2024-05-22 Thread Sergey Nuyanzin (Jira)
[ https://issues.apache.org/jira/browse/FLINK-35423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-35423: Description: After a number of discussions e.g. here [1] It was decided to follow set

[jira] [Created] (FLINK-35423) ARRAY_EXCEPT should support set semantics

2024-05-22 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-35423: --- Summary: ARRAY_EXCEPT should support set semantics Key: FLINK-35423 URL: https://issues.apache.org/jira/browse/FLINK-35423 Project: Flink Issue Type:

[jira] [Comment Edited] (FLINK-16105) Translate "User-defined Sources & Sinks" page of "Table API & SQL" into Chinese

2024-05-22 Thread boxes (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848654#comment-17848654 ] boxes edited comment on FLINK-16105 at 5/22/24 2:57 PM: Hi, Does the

[jira] [Commented] (FLINK-16105) Translate "User-defined Sources & Sinks" page of "Table API & SQL" into Chinese

2024-05-22 Thread boxes (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848654#comment-17848654 ] boxes commented on FLINK-16105: --- Hi, Does the documentation for user-defined sources and sinks still need

Re: [PR] [FLINK-33759] [flink-parquet] Add support for nested array with row/map/array type [flink]

2024-05-22 Thread via GitHub
ViktorCosenza commented on PR #24795: URL: https://github.com/apache/flink/pull/24795#issuecomment-2124999154 > Thanks! The PR looks good to me. Hi @JingsongLi, please also take a look. I think we can port this to fix [apache/paimon#1730](https://github.com/apache/paimon/issues/1730)

[jira] [Commented] (FLINK-34672) HA deadlock between JobMasterServiceLeadershipRunner and DefaultLeaderElectionService

2024-05-22 Thread Matthias Pohl (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848648#comment-17848648 ] Matthias Pohl commented on FLINK-34672: --- I'm still trying to find a reviewer. It's on my plate.

[jira] [Created] (FLINK-35422) Flink dynamically adjusts operator-level resource usage without stopping the job

2024-05-22 Thread Jira
尹博星 created FLINK-35422: --- Summary: Flink dynamically adjusts operator-level resource usage without stopping the job Key: FLINK-35422 URL: https://issues.apache.org/jira/browse/FLINK-35422 Project: Flink

[jira] [Commented] (FLINK-34672) HA deadlock between JobMasterServiceLeadershipRunner and DefaultLeaderElectionService

2024-05-22 Thread Hong Liang Teoh (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848646#comment-17848646 ] Hong Liang Teoh commented on FLINK-34672: - [~mapohl] do we know if this should be a blocker to

[jira] [Comment Edited] (FLINK-34672) HA deadlock between JobMasterServiceLeadershipRunner and DefaultLeaderElectionService

2024-05-22 Thread Hong Liang Teoh (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848646#comment-17848646 ] Hong Liang Teoh edited comment on FLINK-34672 at 5/22/24 2:29 PM: --

[jira] [Commented] (FLINK-34379) table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError

2024-05-22 Thread Hong Liang Teoh (Jira)
[ https://issues.apache.org/jira/browse/FLINK-34379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848644#comment-17848644 ] Hong Liang Teoh commented on FLINK-34379: - [~jeyhunkarimov] Any progress on this patch? >

Re: [PR] [FLINK-35216] Support for RETURNING clause of JSON_QUERY [flink]

2024-05-22 Thread via GitHub
dawidwys commented on code in PR #24704: URL: https://github.com/apache/flink/pull/24704#discussion_r1610035745 ## flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/SqlJsonUtils.java: ## @@ -235,39 +251,85 @@ private static String jsonQuery(

  1   2   3   >