Re: [PR] [FLINK-33687][client] Reuse RestClusterClient in StandaloneClusterDescriptor [flink]

2023-12-13 Thread via GitHub
flinkbot commented on PR #23921: URL: https://github.com/apache/flink/pull/23921#issuecomment-1853773306 ## CI report: * 073a0461b43cc665ecf9c3874273054d59cc2257 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
Jiabao-Sun commented on code in PR #23914: URL: https://github.com/apache/flink/pull/23914#discussion_r1425242859 ## flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/utils/StreamAbstractTestBase.java: ## @@ -22,10 +22,14 @@ import

Re: [PR] [FLINK-33764] Track Heap usage and GC pressure to avoid unnecessary scaling [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
1996fanrui commented on code in PR #726: URL: https://github.com/apache/flink-kubernetes-operator/pull/726#discussion_r1425233692 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/RestApiMetricsCollector.java: ## @@ -78,19 +99,89 @@ protected Map

[jira] [Updated] (FLINK-33687) Reuse RestClusterClient in StandaloneClusterDescriptor to avoid frequent thread create/destroy

2023-12-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33687: --- Labels: pull-request-available (was: ) > Reuse RestClusterClient in

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
Jiabao-Sun commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853767448 Hi @snuyanzin, I think the `SnapshotDirectory.cleanup` may not be the root cause. I debugged it locally, and the directory structure cleared by SnapshotDirectory is start with

[PR] [FLINK-33687][client]Reuse RestClusterClient in StandaloneClusterDescriptor [flink]

2023-12-13 Thread via GitHub
xiangyuf opened a new pull request, #23921: URL: https://github.com/apache/flink/pull/23921 ## What is the purpose of the change Reuse RestClusterClient in StandaloneClusterDescriptor to avoid frequent thread create/destroy. ## Brief change log - Reuse

Re: [PR] [FLINK-33764] Track Heap usage and GC pressure to avoid unnecessary scaling [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
1996fanrui commented on code in PR #726: URL: https://github.com/apache/flink-kubernetes-operator/pull/726#discussion_r1425233692 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/RestApiMetricsCollector.java: ## @@ -78,19 +99,89 @@ protected Map

[jira] [Commented] (FLINK-6755) Allow triggering Checkpoints through command line client

2023-12-13 Thread David Artiga (Jira)
[ https://issues.apache.org/jira/browse/FLINK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796167#comment-17796167 ] David Artiga commented on FLINK-6755: - This is awesome :D > Allow triggering Checkpoints through

Re: [PR] [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem [flink]

2023-12-13 Thread via GitHub
MartijnVisser commented on PR #23489: URL: https://github.com/apache/flink/pull/23489#issuecomment-1853706439 @singhravidutt Can I get your thoughts on https://github.com/apache/flink/pull/23920 as an alternative solution? -- This is an automated message from the Apache Git Service. To

Re: [PR] [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem [flink]

2023-12-13 Thread via GitHub
MartijnVisser commented on code in PR #23489: URL: https://github.com/apache/flink/pull/23489#discussion_r1425195367 ## flink-filesystems/flink-gs-fs-hadoop/pom.xml: ## @@ -204,20 +251,13 @@ under the License.

[jira] [Commented] (FLINK-33312) StreamingFileWriterTest.testCommitFileWhenPartitionIsCommittableByPartitionTime fails on java 21

2023-12-13 Thread Sergey Nuyanzin (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796133#comment-17796133 ] Sergey Nuyanzin commented on FLINK-33312: - No more new occurrences for almost 2 months For that

[jira] [Closed] (FLINK-33312) StreamingFileWriterTest.testCommitFileWhenPartitionIsCommittableByPartitionTime fails on java 21

2023-12-13 Thread Sergey Nuyanzin (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin closed FLINK-33312. --- Resolution: Cannot Reproduce >

[PR] Add Pulsar connector v4.1.0 release [flink-web]

2023-12-13 Thread via GitHub
leonardBang opened a new pull request, #703: URL: https://github.com/apache/flink-web/pull/703 Add Pulsar connector v4.1.0 release -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[jira] [Commented] (FLINK-33804) Add Option to disable showing metrics in JobMananger UI

2023-12-13 Thread Martijn Visser (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796131#comment-17796131 ] Martijn Visser commented on FLINK-33804: [~qqibrow] What is your proposal for the long term

Re: [PR] [FLINK-33813][autoscaler] Improve time formatting for easier reading inside the autoscaler [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
1996fanrui commented on code in PR #731: URL: https://github.com/apache/flink-kubernetes-operator/pull/731#discussion_r1425174343 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/DateTimeUtils.java: ## @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software

Re: [PR] [ FLINK-33603][FileSystems] shade guava in gs-fs filesystem [flink]

2023-12-13 Thread via GitHub
singhravidutt commented on code in PR #23489: URL: https://github.com/apache/flink/pull/23489#discussion_r1425171966 ## flink-filesystems/flink-gs-fs-hadoop/pom.xml: ## @@ -188,6 +212,29 @@ under the License. shade

Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
gyfora commented on code in PR #730: URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425170026 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java: ## @@ -375,9 +376,24 @@ private int

Re: [PR] [FLINK-33813][autoscaler] Improve time formatting for easier reading inside the autoscaler [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
mxm commented on code in PR #731: URL: https://github.com/apache/flink-kubernetes-operator/pull/731#discussion_r1425158303 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/utils/DateTimeUtils.java: ## @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software

Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
mxm commented on code in PR #730: URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425156114 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java: ## @@ -375,9 +376,24 @@ private int

Re: [PR] [FLINK-33812] Autoscale not working properly for standalone application deployment [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
mxm commented on code in PR #730: URL: https://github.com/apache/flink-kubernetes-operator/pull/730#discussion_r1425156114 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/config/FlinkConfigBuilder.java: ## @@ -375,9 +376,24 @@ private int

[jira] [Created] (FLINK-33816) SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain failed due async checkpoint triggering not being completed

2023-12-13 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-33816: - Summary: SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain failed due async checkpoint triggering not being completed Key: FLINK-33816 URL:

[jira] [Updated] (FLINK-33815) Add tests against jdk17 for pulsar connector

2023-12-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33815: --- Labels: pull-request-available (was: ) > Add tests against jdk17 for pulsar connector >

[PR] [FLINK-33815][Connector / Pulsar] Add tests against jdk17 for pulsar connector [flink-connector-pulsar]

2023-12-13 Thread via GitHub
snuyanzin opened a new pull request, #71: URL: https://github.com/apache/flink-connector-pulsar/pull/71 ## Purpose of the change 1.18.x supports ## Brief change log gha ## Verifying this change This change is a trivial rework ##

[jira] [Created] (FLINK-33815) Add tests against jdk17 for pulsar connector

2023-12-13 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-33815: --- Summary: Add tests against jdk17 for pulsar connector Key: FLINK-33815 URL: https://issues.apache.org/jira/browse/FLINK-33815 Project: Flink Issue

Re: [PR] [FLINK-33603][Filesystem] Bundle Guava for GCS connector [flink]

2023-12-13 Thread via GitHub
flinkbot commented on PR #23920: URL: https://github.com/apache/flink/pull/23920#issuecomment-1853622776 ## CI report: * 2ebe7f6e090690486c1954099a2b57283c578192 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[PR] [FLINK-33603][Filesystem] Bundle Guava for GCS connector [flink]

2023-12-13 Thread via GitHub
MartijnVisser opened a new pull request, #23920: URL: https://github.com/apache/flink/pull/23920 ## What is the purpose of the change * Resolve FLINK-33793 `java.lang.NoSuchMethodError when checkpointing in Google Cloud Storage` ## Brief change log * Make sure that

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
snuyanzin commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853578934 > That would mean that we should wait for the snapshot cleanup to finish before calling @AfterEach-annotated methods, wouldn't it? ideally yes -- This is an automated message

Re: [PR] [FLINK-33764] Track Heap usage and GC pressure to avoid unnecessary scaling [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
gyfora commented on PR #726: URL: https://github.com/apache/flink-kubernetes-operator/pull/726#issuecomment-1853576202 > I worry a bit about the implications of enabling this feature by default (also see [#726

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
Jiabao-Sun commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853575930 Thanks @snuyanzin @XComp, I'll try to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
XComp commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853573483 Thanks for this finding, @snuyanzin . That would mean that we should wait for the snapshot cleanup to finish before calling `@AfterEach`-annotated methods, wouldn't it? -- This is an

Re: [PR] [FLINK-33764] Track Heap usage and GC pressure to avoid unnecessary scaling [flink-kubernetes-operator]

2023-12-13 Thread via GitHub
gyfora commented on code in PR #726: URL: https://github.com/apache/flink-kubernetes-operator/pull/726#discussion_r1425082293 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/metrics/ScalingMetric.java: ## @@ -65,15 +69,35 @@ public enum ScalingMetric {

Re: [PR] [FLINK-33588][Flink-Runtime] Fix NullArgumentException of getQuantile method in DescriptiveStatisticsHistogramStatistics [flink]

2023-12-13 Thread via GitHub
JingGe commented on PR #23915: URL: https://github.com/apache/flink/pull/23915#issuecomment-1853555679 Thanks @zhutong6688 for driving it. Could you squash the commits? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
snuyanzin commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853554316 @Jiabao-Sun , @XComp it seems I found the reason junit5.10.1 makes it always failing and it is becoming a bit more clear there are 2 threads 1. junit5 trying to

Re: [PR] [FLINK-33730][doc] update the Flink upgrade savepoint compatibility table doc [flink]

2023-12-13 Thread via GitHub
JingGe merged PR #23859: URL: https://github.com/apache/flink/pull/23859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [FLINK-33730][doc] update the Flink upgrade savepoint compatibility table doc [flink]

2023-12-13 Thread via GitHub
JingGe commented on PR #23859: URL: https://github.com/apache/flink/pull/23859#issuecomment-1853547424 Thanks @PatrickRen for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [FLINK-33641][test] Suppress the DirectoryNotEmptyException in StreamingWithStateTestBase to prevent test failures [flink]

2023-12-13 Thread via GitHub
XComp commented on PR #23914: URL: https://github.com/apache/flink/pull/23914#issuecomment-1853544386 > The var tempFolder: Path = _ defines in scala is an object field already, for each @Test method, it is different. Thanks for clarification. My lack of Scala knowledge made me mix

Re: [PR] [FLINK-32028][connectors/elasticsearch] Allow customising bulk failure handling [flink-connector-elasticsearch]

2023-12-13 Thread via GitHub
schulzp commented on PR #83: URL: https://github.com/apache/flink-connector-elasticsearch/pull/83#issuecomment-1853540370 @reswqa, there are two things that need to be fixed: 1. use `MetricsGroupTestUtils#mockWriterMetricGroup()` instead of `InternalSinkWriterMetricGroup.mock()`,

[jira] [Updated] (FLINK-31752) SourceOperatorStreamTask increments numRecordsOut twice

2023-12-13 Thread Yun Tang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang updated FLINK-31752: - Fix Version/s: 1.17.1 > SourceOperatorStreamTask increments numRecordsOut twice >

[jira] [Updated] (FLINK-31752) SourceOperatorStreamTask increments numRecordsOut twice

2023-12-13 Thread Yun Tang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang updated FLINK-31752: - Fix Version/s: 1.18.0 > SourceOperatorStreamTask increments numRecordsOut twice >

[jira] [Commented] (FLINK-33801) Revert to last scaling when job becomes unhealthy after scaling

2023-12-13 Thread Gyula Fora (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796082#comment-17796082 ] Gyula Fora commented on FLINK-33801: This is basically similar to the rollback feature in the

[jira] [Commented] (FLINK-33508) Support for wildcard paths in Flink History Server for multi cluster environment

2023-12-13 Thread Anika Kelhanka (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796058#comment-17796058 ] Anika Kelhanka commented on FLINK-33508: Approach to update Flink's History Server logic to

Re: [PR] make sure that super.close() must be executed! [flink-connector-kafka]

2023-12-13 Thread via GitHub
boring-cyborg[bot] commented on PR #74: URL: https://github.com/apache/flink-connector-kafka/pull/74#issuecomment-1853442445 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is

[jira] [Created] (FLINK-33814) Autoscaler Standalone control loop supports multiple thread

2023-12-13 Thread Rui Fan (Jira)
Rui Fan created FLINK-33814: --- Summary: Autoscaler Standalone control loop supports multiple thread Key: FLINK-33814 URL: https://issues.apache.org/jira/browse/FLINK-33814 Project: Flink Issue

<    1   2