[jira] [Commented] (FLINK-25233) UpsertKafkaTableITCase.testAggregate fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513190#comment-17513190
]
Yun Gao commented on FLINK-25233:
-
KafkaTableITCase@1.14:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33773&view=logs&j=4be4ed2b-549a-533d-aa33-09e28e360cc8&t=f7d83ad5-3324-5307-0eb3-819065cdcb38&l=8443
> UpsertKafkaTableITCase.testAggregate fails on AZP
> -
>
> Key: FLINK-25233
> URL: https://issues.apache.org/jira/browse/FLINK-25233
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.16.0
>
>
> {{UpsertKafkaTableITCase.testAggregate}} fails on AZP with
> {code}
> 2021-12-09T01:41:49.8038402Z Dec 09 01:41:49 [ERROR]
> UpsertKafkaTableITCase.testAggregate Time elapsed: 90.624 s <<< ERROR!
> 2021-12-09T01:41:49.8039372Z Dec 09 01:41:49
> java.util.concurrent.ExecutionException:
> org.apache.flink.table.api.TableException: Failed to wait job finish
> 2021-12-09T01:41:49.8040303Z Dec 09 01:41:49 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2021-12-09T01:41:49.8040956Z Dec 09 01:41:49 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2021-12-09T01:41:49.8041862Z Dec 09 01:41:49 at
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
> 2021-12-09T01:41:49.8042939Z Dec 09 01:41:49 at
> org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
> 2021-12-09T01:41:49.8044130Z Dec 09 01:41:49 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.wordCountToUpsertKafka(UpsertKafkaTableITCase.java:436)
> 2021-12-09T01:41:49.8045308Z Dec 09 01:41:49 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testAggregate(UpsertKafkaTableITCase.java:79)
> 2021-12-09T01:41:49.8045940Z Dec 09 01:41:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-12-09T01:41:49.8052892Z Dec 09 01:41:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-12-09T01:41:49.8053812Z Dec 09 01:41:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-09T01:41:49.8054458Z Dec 09 01:41:49 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-12-09T01:41:49.8055027Z Dec 09 01:41:49 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-09T01:41:49.8055649Z Dec 09 01:41:49 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-09T01:41:49.8056644Z Dec 09 01:41:49 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-09T01:41:49.8057911Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-09T01:41:49.8058858Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-12-09T01:41:49.8059907Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-12-09T01:41:49.8060871Z Dec 09 01:41:49 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-12-09T01:41:49.8061847Z Dec 09 01:41:49 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-09T01:41:49.8062898Z Dec 09 01:41:49 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-09T01:41:49.8063804Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-09T01:41:49.8064963Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-12-09T01:41:49.8065992Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-12-09T01:41:49.8066940Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-12-09T01:41:49.8067939Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-12-09T01:41:49.8068904Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-09T01:41:49.8069837Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-09T01:41:49.8070715Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-12-09T01:41:49.8071587Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-12-09T01:
[jira] [Assigned] (FLINK-26879) LogStartupMode.LATEST does not work
[ https://issues.apache.org/jira/browse/FLINK-26879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-26879: Assignee: Jane Chan > LogStartupMode.LATEST does not work > --- > > Key: FLINK-26879 > URL: https://issues.apache.org/jira/browse/FLINK-26879 > Project: Flink > Issue Type: Bug > Components: Table Store >Affects Versions: table-store-0.1.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Blocker > Fix For: table-store-0.1.0 > > > We should avoid creating a hybrid source when the latest mode is enabled, > o.w. we cannot consume changelog from log store. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-18757) FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed with "Topic XX not present in metadata after 60000 ms"
[
https://issues.apache.org/jira/browse/FLINK-18757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao closed FLINK-18757.
---
Resolution: Duplicate
> FlinkKafkaProducerITCase.testMigrateFromAtLeastOnceToExactlyOnce failed with
> "Topic XX not present in metadata after 6 ms"
> --
>
> Key: FLINK-18757
> URL: https://issues.apache.org/jira/browse/FLINK-18757
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: Fabian Paul
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> stale-assigned
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5024&view=logs&j=1fc6e7bf-633c-5081-c32a-9dea24b05730&t=0d9ad4c1-5629-5ffc-10dc-113ca91e23c5
> {code}
> 2020-07-29T21:14:51.9829834Z Test
> testMigrateFromAtLeastOnceToExactlyOnce(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> failed with:
> 2020-07-29T21:14:51.9830447Z org.apache.kafka.common.errors.TimeoutException:
> Topic testMigrateFromAtLeastOnceToExactlyOnce not present in metadata after
> 6 ms.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26721) PulsarSourceITCase.testSavepoint failed on azure pipeline
[
https://issues.apache.org/jira/browse/FLINK-26721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513192#comment-17513192
]
Yun Gao commented on FLINK-26721:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33772&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=26502
> PulsarSourceITCase.testSavepoint failed on azure pipeline
> -
>
> Key: FLINK-26721
> URL: https://issues.apache.org/jira/browse/FLINK-26721
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0
>Reporter: Yun Gao
>Priority: Major
> Labels: build-stability
>
> {code:java}
> Mar 18 05:49:52 [ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 315.581 s <<< FAILURE! - in
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase
> Mar 18 05:49:52 [ERROR]
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase.testSavepoint(TestEnvironment,
> DataStreamSourceExternalContext, CheckpointingMode)[1] Time elapsed:
> 140.803 s <<< FAILURE!
> Mar 18 05:49:52 java.lang.AssertionError:
> Mar 18 05:49:52
> Mar 18 05:49:52 Expecting
> Mar 18 05:49:52
> Mar 18 05:49:52 to be completed within 2M.
> Mar 18 05:49:52
> Mar 18 05:49:52 exception caught while trying to get the future result:
> java.util.concurrent.TimeoutException
> Mar 18 05:49:52 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> Mar 18 05:49:52 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> Mar 18 05:49:52 at
> org.assertj.core.internal.Futures.assertSucceededWithin(Futures.java:109)
> Mar 18 05:49:52 at
> org.assertj.core.api.AbstractCompletableFutureAssert.internalSucceedsWithin(AbstractCompletableFutureAssert.java:400)
> Mar 18 05:49:52 at
> org.assertj.core.api.AbstractCompletableFutureAssert.succeedsWithin(AbstractCompletableFutureAssert.java:396)
> Mar 18 05:49:52 at
> org.apache.flink.connector.testframe.testsuites.SourceTestSuiteBase.checkResultWithSemantic(SourceTestSuiteBase.java:766)
> Mar 18 05:49:52 at
> org.apache.flink.connector.testframe.testsuites.SourceTestSuiteBase.restartFromSavepoint(SourceTestSuiteBase.java:399)
> Mar 18 05:49:52 at
> org.apache.flink.connector.testframe.testsuites.SourceTestSuiteBase.testSavepoint(SourceTestSuiteBase.java:241)
> Mar 18 05:49:52 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Mar 18 05:49:52 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Mar 18 05:49:52 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Mar 18 05:49:52 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 18 05:49:52 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> Mar 18 05:49:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$inv
[jira] [Updated] (FLINK-25450) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25450:
Priority: Major (was: Minor)
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on
> azure
> -
>
> Key: FLINK-25450
> URL: https://issues.apache.org/jira/browse/FLINK-25450
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Dec 25 03:59:33 [ERROR] Errors:
> Dec 25 03:59:33 [ERROR]
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout:1747->runJobFailureWhenTaskExecutorTerminatesTest:1875
> » Execution
> Dec 25 03:59:33 [INFO]
> Dec 25 03:59:33 [ERROR] Tests run: 5960, Failures: 0, Errors: 1, Skipped: 26
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28590&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=8610
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-26875) Operator Helm: Extends the rbac options with separate namespaces param
[
https://issues.apache.org/jira/browse/FLINK-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xin Hao resolved FLINK-26875.
-
Resolution: Not A Problem
> Operator Helm: Extends the rbac options with separate namespaces param
> --
>
> Key: FLINK-26875
> URL: https://issues.apache.org/jira/browse/FLINK-26875
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Xin Hao
>Priority: Minor
>
> Currently, the Flink Kubernetes Operator Helm Chart uses the
> *watchNamespaces* to determine which namespaces should the roles be created.
> {code:java}
> {{- if .Values.rbac.create }}
> {{- if .Values.watchNamespaces }}
> {{- range .Values.watchNamespaces }}
> ...{code}
> But my use case is that I have another operator for myself to observe the
> Flink Operator's CRs so that I want to create roles in more namespaces.
> So the requirement for me is that add another param called namespaces under
> the *rbac* section and *defaults to watchNamespaces.*
> It will look like this:
> {code:java}
> # values.yaml
> rbac:
> create: true
> # The namespaces should create the roles, defaults to watchNamespaces
> # namespaces: ["flink"]
> # rbac.yaml
> {{- if or .Values.rbac.namespaces .Values.watchNamespaces }}
> {{- range (default .Values.watchNamespaces .Values.rbac.namespaces) }}{code}
> So that the users can create their role bindings.
> The reason that the users shouldn't create roles by themselves is the
> contents of the role should be managed y the Flink Helm chart.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25450) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25450:
Affects Version/s: 1.16.0
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on
> azure
> -
>
> Key: FLINK-25450
> URL: https://issues.apache.org/jira/browse/FLINK-25450
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Yun Gao
>Priority: Major
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Dec 25 03:59:33 [ERROR] Errors:
> Dec 25 03:59:33 [ERROR]
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout:1747->runJobFailureWhenTaskExecutorTerminatesTest:1875
> » Execution
> Dec 25 03:59:33 [INFO]
> Dec 25 03:59:33 [ERROR] Tests run: 5960, Failures: 0, Errors: 1, Skipped: 26
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28590&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=8610
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25450) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513193#comment-17513193
]
Yun Gao commented on FLINK-25450:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33772&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=8352
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on
> azure
> -
>
> Key: FLINK-25450
> URL: https://issues.apache.org/jira/browse/FLINK-25450
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Dec 25 03:59:33 [ERROR] Errors:
> Dec 25 03:59:33 [ERROR]
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout:1747->runJobFailureWhenTaskExecutorTerminatesTest:1875
> » Execution
> Dec 25 03:59:33 [INFO]
> Dec 25 03:59:33 [ERROR] Tests run: 5960, Failures: 0, Errors: 1, Skipped: 26
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28590&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=8610
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25450) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25450:
Labels: test-stability (was: auto-deprioritized-major test-stability)
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on
> azure
> -
>
> Key: FLINK-25450
> URL: https://issues.apache.org/jira/browse/FLINK-25450
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Dec 25 03:59:33 [ERROR] Errors:
> Dec 25 03:59:33 [ERROR]
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout:1747->runJobFailureWhenTaskExecutorTerminatesTest:1875
> » Execution
> Dec 25 03:59:33 [INFO]
> Dec 25 03:59:33 [ERROR] Tests run: 5960, Failures: 0, Errors: 1, Skipped: 26
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28590&view=logs&j=77a9d8e1-d610-59b3-fc2a-4766541e0e33&t=125e07e7-8de0-5c6c-a541-a567415af3ef&l=8610
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26852) RocksDBMapState#clear not forwarding exceptions
[
https://issues.apache.org/jira/browse/FLINK-26852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513195#comment-17513195
]
Yun Tang commented on FLINK-26852:
--
I think this is a really bug, asked [~masteryhx] to take this ticket.
> RocksDBMapState#clear not forwarding exceptions
> ---
>
> Key: FLINK-26852
> URL: https://issues.apache.org/jira/browse/FLINK-26852
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.15.0, 1.14.4
>Reporter: Nico Kruber
>Priority: Major
>
> I accidentally found an inconsistent behaviour in the RocksDB state backend
> implementation:
> If there's an exception in {{AbstractRocksDBState#clear()}} it will forward
> that inside a {{FlinkRuntimeException}}.
> However, if there's an exception in {{RocksDBMapState#clear}} it will merely
> print the exception stacktrace and continue as is.
> I assume, forwarding the exception as {{FlinkRuntimeException}} should be the
> desired behaviour for both use cases...
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-26852) RocksDBMapState#clear not forwarding exceptions
[
https://issues.apache.org/jira/browse/FLINK-26852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang reassigned FLINK-26852:
Assignee: Hangxiang Yu
> RocksDBMapState#clear not forwarding exceptions
> ---
>
> Key: FLINK-26852
> URL: https://issues.apache.org/jira/browse/FLINK-26852
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.15.0, 1.14.4
>Reporter: Nico Kruber
>Assignee: Hangxiang Yu
>Priority: Major
>
> I accidentally found an inconsistent behaviour in the RocksDB state backend
> implementation:
> If there's an exception in {{AbstractRocksDBState#clear()}} it will forward
> that inside a {{FlinkRuntimeException}}.
> However, if there's an exception in {{RocksDBMapState#clear}} it will merely
> print the exception stacktrace and continue as is.
> I assume, forwarding the exception as {{FlinkRuntimeException}} should be the
> desired behaviour for both use cases...
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zoltar9264 opened a new pull request #19252: [FLINK-25867][docs] translate ChangelogBackend documentation to chinese
zoltar9264 opened a new pull request #19252: URL: https://github.com/apache/flink/pull/19252 ## What is the purpose of the change Translate ChangelogBackend documentation to chinese. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes (internal: flink-docs -> flink-dstl-dfs) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (see above) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19252: [FLINK-25867][docs] translate ChangelogBackend documentation to chinese
zoltar9264 commented on pull request #19252: URL: https://github.com/apache/flink/pull/19252#issuecomment-1080276308 Hi @curcur , can you help me review this pull request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19252: [FLINK-25867][docs] translate ChangelogBackend documentation to chinese
flinkbot commented on pull request #19252: URL: https://github.com/apache/flink/pull/19252#issuecomment-1080278121 ## CI report: * 29c3421f68005d60fd11aee2ee047e08e3b81ec0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #71: [FLINK-26443] Add benchmark framework
lindong28 commented on a change in pull request #71:
URL: https://github.com/apache/flink-ml/pull/71#discussion_r836029049
##
File path: flink-ml-benchmark/README.md
##
@@ -0,0 +1,261 @@
+# Flink ML Benchmark Guideline
+
+This document provides instructions about how to run benchmarks on Flink ML's
+stages.
+
+## Write Benchmark Programs
+
+### Add Maven Dependencies
+
+In order to write Flink ML's java benchmark programs, first make sure that the
+following dependencies have been added to your maven project's `pom.xml`.
+
+```xml
+
+ org.apache.flink
+ flink-ml-core_${scala.binary.version}
+ ${flink.ml.version}
+
+
+
+ org.apache.flink
+ flink-ml-iteration_${scala.binary.version}
+ ${flink.ml.version}
+
+
+
+ org.apache.flink
+ flink-ml-lib_${scala.binary.version}
+ ${flink.ml.version}
+
+
+
+ org.apache.flink
+ flink-ml-benchmark_${scala.binary.version}
+ ${flink.ml.version}
+
+
+
+ org.apache.flink
+ statefun-flink-core
+ 3.1.0
+
+
+ org.apache.flink
+ flink-streaming-java_2.12
+
+
+
+
+
+ org.apache.flink
+ flink-streaming-java_${scala.binary.version}
+ ${flink.version}
+
+
+
+ org.apache.flink
+ flink-table-api-java-bridge_${scala.binary.version}
+ ${flink.version}
+
+
+
+ org.apache.flink
+ flink-table-planner_${scala.binary.version}
+ ${flink.version}
+
+
+
+ org.apache.flink
+ flink-clients_${scala.binary.version}
+ ${flink.version}
+
+```
+
+### Write Java Program
+
+Then you can write a program as follows to run benchmark on Flink ML stages.
The
+example code below tests the performance of Flink ML's KMeans algorithm, with
+the default configuration parameters used.
+
+```java
+public class Main {
+public static void main(String[] args) throws Exception {
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+KMeans kMeans = new KMeans();
+KMeansInputsGenerator inputsGenerator = new KMeansInputsGenerator();
+
+BenchmarkResult result =
+BenchmarkUtils.runBenchmark("exampleBenchmark", tEnv, kMeans,
inputsGenerator);
+
+BenchmarkUtils.printResult(result);
+}
+}
+```
+
+### Execute Benchmark Program
+
+After executing the `main()` method above, you will see benchmark results
+printed out in your terminal. An example of the printed content is as follows.
+
+```
+Benchmark Name: exampleBenchmark
+Total Execution Time(ms): 828.0
+```
+
+### Configure Benchmark Parameters
+
+If you want to run benchmark on customed configuration parameters, you can set
+them with Flink ML's `WithParams` API as follows.
+
+```java
+KMeans kMeans = new KMeans()
+ .setK(5)
+ .setMaxIter(50);
+KMeansInputsGenerator inputsGenerator = new KMeansInputsGenerator()
+ .setDims(3)
+ .setDataSize(1);
+```
+
+## Execute Benchmark through Command-Line Interface (CLI)
+
+You can also configure and execute benchmarks through Command-Line Interface
+(CLI) without writing java programs.
+
+### Prerequisites
+
+Before using Flink ML's CLI, make sure you have installed Flink 1.14 in your
+local environment, and that you have started a Flink cluster locally. If not,
+you can start a standalone session with the following command.
+
+```bash
+$ start-cluster
+```
+
+In order to use Flink ML's CLI you need to have the latest binary distribution
+of Flink ML. You can acquire the distribution by building Flink ML's source
code
+locally, which means to execute the following command in Flink ML repository's
+root directory.
+
+```bash
+$ mvn clean package -DskipTests
+```
+
+After executing the command above, you will be able to find the binary
+distribution under
+`./flink-ml-dist/target/flink-ml--bin/flink-ml-/`.
+
+### Run Benchmark CLI
+
+In the binary distribution's folder, execute the following command to run an
+example benchmark.
+
+```bash
+$ ./bin/flink-ml-benchmark.sh ./examples/benchmark-example-conf.json
+```
+
+You will notice that some Flink job is submitted to your Flink cluster, and the
+following information is printed out in your terminal. This means that you have
+successfully executed a benchmark on `KMeansModel`.
+
+```
+Job has been submitted with JobID 85b4a33df5c00a315e0d1142e1d743be
+Program execution finished
+Job with JobID 85b4a33df5c00a315e0d1142e1d743be has finished.
+Job Runtime: 828 ms
+
+Benchmark Name: KMeansModel-1
+Total Execution Time(ms): 828.0
+
+```
+
+### Save Benchmark Result to File
+
+`flink-ml-benchmark.sh` has redirected all warnings and process logs to stderr,
+and the benchmark results to stdout. So if you write the command in the
+following way
+
+```bash
+$ ./bin/flink-ml-benchmark.sh ./examples/benchmark-example-conf.json >
output.txt
Review comment:
This command is almost the same as the command above.
Would it be simpler to just have one section `Run Benchmark CLI`? And in
this section, we can explain how to get and interpret
[GitHub] [flink] flinkbot edited a comment on pull request #19252: [FLINK-25867][docs] translate ChangelogBackend documentation to chinese
flinkbot edited a comment on pull request #19252: URL: https://github.com/apache/flink/pull/19252#issuecomment-1080278121 ## CI report: * 29c3421f68005d60fd11aee2ee047e08e3b81ec0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33805) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23890) CepOperator may create a large number of timers and cause performance problems
[
https://issues.apache.org/jira/browse/FLINK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513199#comment-17513199
]
Nicholas Jiang commented on FLINK-23890:
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to
currentWatermark+windowTime，not +1. Do you have interest to contribute a pull
request to improve?
> CepOperator may create a large number of timers and cause performance problems
> --
>
> Key: FLINK-23890
> URL: https://issues.apache.org/jira/browse/FLINK-23890
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.12.1
>Reporter: Yue Ma
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-08-20-13-59-05-977.png
>
>
> There are two situations in the CepOperator that may register the time when
> dealing with EventTime.
> when the processElement will buffer the data first, and then register a timer
> with a timestamp of watermark+1.
> {code:java}
> if (timestamp > timerService.currentWatermark()) {
> // we have an event with a valid timestamp, so
> // we buffer it until we receive the proper watermark.
> saveRegisterWatermarkTimer();
> bufferEvent(value, timestamp);
> }{code}
> The other is when the EventTimer is triggered, if sortedTimestamps or
> partialMatches are not empty, a timer will also be registered.
> {code:java}
> if (!sortedTimestamps.isEmpty() || !partialMatches.isEmpty()) {
> saveRegisterWatermarkTimer();
> }{code}
>
> The problem is, if the partialMatches corresponding to each of my keys are
> not empty. Then every time the watermark advances, the timers of all keys
> will be triggered, and then a new EventTimer is re-registered under each key.
> When the number of task keys is very large, this operation greatly affects
> performance.
> !https://code.byted.org/inf/flink/uploads/91aee639553df07fa376cf2865e91fd2/image.png!
> I think it is unnecessary to register EventTimer frequently like this and can
> we make the following changes?
> When an event comes, the timestamp of the EventTimer we registered is equal
> to the EventTime of this event instead of watermark + 1.
> When a new ComputionState with window is created (like *withIn* pattern ),
> we use the timeout of this window to create EventTimer (EventTime +
> WindowTime).
> After making such an attempt in our test environment, the number of
> registered timers has been greatly reduced, and the performance has been
> greatly improved.
> !https://code.byted.org/inf/flink/uploads/24b85492c6a34a35c4445a4fd46c8363/image.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-23890) CepOperator may create a large number of timers and cause performance problems
[
https://issues.apache.org/jira/browse/FLINK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513199#comment-17513199
]
Nicholas Jiang edited comment on FLINK-23890 at 3/28/22, 7:21 AM:
--
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to
currentWatermark+windowTime, not +1.
BTW, do you have interest to contribute a pull request to improve?
was (Author: nicholasjiang):
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to
currentWatermark+windowTime，not +1. Do you have interest to contribute a pull
request to improve?
> CepOperator may create a large number of timers and cause performance problems
> --
>
> Key: FLINK-23890
> URL: https://issues.apache.org/jira/browse/FLINK-23890
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.12.1
>Reporter: Yue Ma
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-08-20-13-59-05-977.png
>
>
> There are two situations in the CepOperator that may register the time when
> dealing with EventTime.
> when the processElement will buffer the data first, and then register a timer
> with a timestamp of watermark+1.
> {code:java}
> if (timestamp > timerService.currentWatermark()) {
> // we have an event with a valid timestamp, so
> // we buffer it until we receive the proper watermark.
> saveRegisterWatermarkTimer();
> bufferEvent(value, timestamp);
> }{code}
> The other is when the EventTimer is triggered, if sortedTimestamps or
> partialMatches are not empty, a timer will also be registered.
> {code:java}
> if (!sortedTimestamps.isEmpty() || !partialMatches.isEmpty()) {
> saveRegisterWatermarkTimer();
> }{code}
>
> The problem is, if the partialMatches corresponding to each of my keys are
> not empty. Then every time the watermark advances, the timers of all keys
> will be triggered, and then a new EventTimer is re-registered under each key.
> When the number of task keys is very large, this operation greatly affects
> performance.
> !https://code.byted.org/inf/flink/uploads/91aee639553df07fa376cf2865e91fd2/image.png!
> I think it is unnecessary to register EventTimer frequently like this and can
> we make the following changes?
> When an event comes, the timestamp of the EventTimer we registered is equal
> to the EventTime of this event instead of watermark + 1.
> When a new ComputionState with window is created (like *withIn* pattern ),
> we use the timeout of this window to create EventTimer (EventTime +
> WindowTime).
> After making such an attempt in our test environment, the number of
> registered timers has been greatly reduced, and the performance has been
> greatly improved.
> !https://code.byted.org/inf/flink/uploads/24b85492c6a34a35c4445a4fd46c8363/image.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 743339c07c0549f2774df718cf728058f30174fe Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33784) * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) * 016eace10679897b617be44dd47402c362b7d564 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22888) Matches results may be wrong when using notNext as the last part of the pattern with Window
[
https://issues.apache.org/jira/browse/FLINK-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Jiang updated FLINK-22888:
---
Description:
the pattern is like
Pattern.begin("start").where(records == "a")
.notNext("notNext").where(records == "b")
.withIn(5milliseconds).
If there is only one event *"a"* in 5 milliseconds. I think this *“a”* should
be output as the correct result of the match next time in advanceTime.
But in the actual operation of CEP. This “a” will be treated as matching
timeout data
{code:java}
// code placeholder
@Test
public void testNoNextWithWindow() throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// (Event, timestamp)
DataStream input = env.fromElements(
Tuple2.of(new Event(1, "start", 1.0), 5L),
// last element for high final watermark
Tuple2.of(new Event(5, "final", 5.0), 100L)
).assignTimestampsAndWatermarks(new
AssignerWithPunctuatedWatermarks>() {
@Override
public long extractTimestamp(Tuple2 element, long
previousTimestamp) {
return element.f1;
}
@Override
public Watermark checkAndGetNextWatermark(Tuple2
lastElement, long extractedTimestamp) {
return new Watermark(lastElement.f1 - 5);
}
}).map(new MapFunction, Event>() {
@Override
public Event map(Tuple2 value) throws Exception {
return value.f0;
}
});
Pattern pattern = Pattern.begin("start").where(new
SimpleCondition() {
@Override
public boolean filter(Event value) throws Exception {
return value.getName().equals("start");
}
}).notNext("middle").where(new SimpleCondition() {
@Override
public boolean filter(Event value) throws Exception {
return value.getName().equals("middle");
}
}).within(Time.milliseconds(5L));
DataStream result = CEP.pattern(input, pattern).select(
new PatternSelectFunction() {
@Override
public String select(Map> pattern) {
StringBuilder builder = new StringBuilder();
builder.append(pattern.get("start").get(0).getId());
return builder.toString();
}
}
);
List resultList = new ArrayList<>();
DataStreamUtils.collect(result).forEachRemaining(resultList::add);
resultList.sort(String::compareTo);
assertEquals(Arrays.asList("1"), resultList);
}
{code}
was:
the pattern is like
Pattern.begin("start").where(records == "a")
.notNext("notNext").where(records == "b")
.withIn(5milliseconds).
If there is only one event *"a"* in 5 milliseconds. I think this *“a”* should
be output as the correct result of the match next time in advanceTime.
But in the actual operation of CEP. This “a” will be treated as matching
timeout data
{code:java}
// code placeholder
@Test
public void testNoNextWithWindow() throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// (Event, timestamp)
DataStream input = env.fromElements(
Tuple2.of(new Event(1, "start", 1.0), 5L),
// last element for high final watermark
Tuple2.of(new Event(5, "final", 5.0), 100L)
).assignTimestampsAndWatermarks(new
AssignerWithPunctuatedWatermarks>() {
@Override
public long extractTimestamp(Tuple2 element, long
previousTimestamp) {
return element.f1;
}
@Override
public Watermark checkAndGetNextWatermark(Tuple2
lastElement, long extractedTimestamp) {
return new Watermark(lastElement.f1 - 5);
}
}).map(new MapFunction, Event>() {
@Override
public Event map(Tuple2 value) throws Exception {
return value.f0;
}
});
Pattern pattern = Pattern.begin("start").where(new
SimpleCondition() {
@Override
public boolean filter(Event value) throws Exception {
return value.getName().equals("start");
}
}).notNext("middle").where(new SimpleCondition() {
@Override
public boolean filter(Event value) throws Exception {
return value.getName().equals("middle");
}
}).within(Time.milliseconds(5L));
DataStream result = CEP.pattern(input, pattern).select(
new PatternSelectFunction() {
@Override
public String select(Map> pattern) {
StringBuilder builder = new StringBuilder();
builder.append(pattern.get("start").get(0).getId());
return builder.toString();
}
}
);
List resultList = new ArrayList<>();
DataStreamUtils.collect(result).forEachRemaining(resultList::add);
resultList.sort(String::compareTo);
assertEquals(Arrays.asList("1"), resultList);
}
{co
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) * 23e0ae1ec36e42ad5b003c4693805cc90d4f3b8c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23890) CepOperator may create a large number of timers and cause performance problems
[
https://issues.apache.org/jira/browse/FLINK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513199#comment-17513199
]
Nicholas Jiang edited comment on FLINK-23890 at 3/28/22, 7:32 AM:
--
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to EventTime+WindowTime,
not +1.
BTW, do you have interest to contribute a pull request to improve?
was (Author: nicholasjiang):
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to
currentWatermark+windowTime, not +1.
BTW, do you have interest to contribute a pull request to improve?
> CepOperator may create a large number of timers and cause performance problems
> --
>
> Key: FLINK-23890
> URL: https://issues.apache.org/jira/browse/FLINK-23890
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.12.1
>Reporter: Yue Ma
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-08-20-13-59-05-977.png
>
>
> There are two situations in the CepOperator that may register the time when
> dealing with EventTime.
> when the processElement will buffer the data first, and then register a timer
> with a timestamp of watermark+1.
> {code:java}
> if (timestamp > timerService.currentWatermark()) {
> // we have an event with a valid timestamp, so
> // we buffer it until we receive the proper watermark.
> saveRegisterWatermarkTimer();
> bufferEvent(value, timestamp);
> }{code}
> The other is when the EventTimer is triggered, if sortedTimestamps or
> partialMatches are not empty, a timer will also be registered.
> {code:java}
> if (!sortedTimestamps.isEmpty() || !partialMatches.isEmpty()) {
> saveRegisterWatermarkTimer();
> }{code}
>
> The problem is, if the partialMatches corresponding to each of my keys are
> not empty. Then every time the watermark advances, the timers of all keys
> will be triggered, and then a new EventTimer is re-registered under each key.
> When the number of task keys is very large, this operation greatly affects
> performance.
> !https://code.byted.org/inf/flink/uploads/91aee639553df07fa376cf2865e91fd2/image.png!
> I think it is unnecessary to register EventTimer frequently like this and can
> we make the following changes?
> When an event comes, the timestamp of the EventTimer we registered is equal
> to the EventTime of this event instead of watermark + 1.
> When a new ComputionState with window is created (like *withIn* pattern ),
> we use the timeout of this window to create EventTimer (EventTime +
> WindowTime).
> After making such an attempt in our test environment, the number of
> registered timers has been greatly reduced, and the performance has been
> greatly improved.
> !https://code.byted.org/inf/flink/uploads/24b85492c6a34a35c4445a4fd46c8363/image.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) * e4b1d6908ea40c3008c237b1d41fd9b2d8726d47 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration
Yang Wang created FLINK-26881:
-
Summary: Stop with savepoint should pick up the targetDirectory
from Flink configuration
Key: FLINK-26881
URL: https://issues.apache.org/jira/browse/FLINK-26881
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Yang Wang
Fix For: kubernetes-operator-0.1.0
Upgrading stateless FlinkDeployment to savepoint still could not work. Because
{{FlinkService#cancelJob}} does not pick up the target directory in the flink
configuration.
{code:java}
case SAVEPOINT:
String savepoint =
clusterClient
.stopWithSavepoint(jobID, false, null)
.get(1, TimeUnit.MINUTES); {code}
We should use configured savepoint directory instead of {{null}} in the above
implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26424) RequestTimeoutException
[ https://issues.apache.org/jira/browse/FLINK-26424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513212#comment-17513212 ] Till Rohrmann commented on FLINK-26424: --- Yes, such a feature will definitely be an opt-in feature. Thanks for your feedback [~Fil Karnicki]. > RequestTimeoutException > > > Key: FLINK-26424 > URL: https://issues.apache.org/jira/browse/FLINK-26424 > Project: Flink > Issue Type: Bug > Components: Stateful Functions >Affects Versions: statefun-3.2.0 >Reporter: xinchenyuan >Priority: Major > > there is no max retries, all I got is the call timeout > as doc said, [Transport Spec > |https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/docs/modules/http-endpoint/#transport-1]call > will be failed after timeout. > but when expcetion raised, runtime restart, I'm confused why a function > internal error will cause such a big problem, will MAX RETRIES be a > configurable param? > > 2022-02-28 17:58:32 > org.apache.flink.statefun.flink.core.functions.StatefulFunctionInvocationException: > An error occurred when attempting to invoke function FunctionType(tendoc, > AlertNotificationIngressCkafka). > at > org.apache.flink.statefun.flink.core.functions.StatefulFunction.receive(StatefulFunction.java:50) > at > org.apache.flink.statefun.flink.core.functions.ReusableContext.apply(ReusableContext.java:73) > at > org.apache.flink.statefun.flink.core.functions.FunctionActivation.applyNextPendingEnvelope(FunctionActivation.java:50) > at > org.apache.flink.statefun.flink.core.functions.LocalFunctionGroup.processNextEnvelope(LocalFunctionGroup.java:61) > at > org.apache.flink.statefun.flink.core.functions.Reductions.processEnvelopes(Reductions.java:164) > at > org.apache.flink.statefun.flink.core.functions.AsyncSink.drainOnOperatorThread(AsyncSink.java:119) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) > at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl.yield(MailboxExecutorImpl.java:86) > at > org.apache.flink.statefun.flink.core.functions.FunctionGroupOperator.processElement(FunctionGroupOperator.java:88) > at > org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:108) > at > org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:89) > at > org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) > at > org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) > at > org.apache.flink.statefun.flink.core.feedback.FeedbackUnionOperator.sendDownstream(FeedbackUnionOperator.java:186) > at > org.apache.flink.statefun.flink.core.feedback.FeedbackUnionOperator.processElement(FeedbackUnionOperator.java:86) > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) > at > org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) > at > org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) > at > org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) > at java.base/java.lang.Thread.run(Unknown Source) > Caused by: java.lang.IllegalStateException: Failure forwarding a message to a > remote function Address(tendoc, AlertNotificationIngressCkafka, cls-message) > at > org.apache.flink.statefun.flink.core.reqreply.RequestReplyFunction.onAsyncResult(RequestReplyFunction.java:170) > at > org.apache.flink.statefun.flink.core.reqreply.RequestReplyFunction.invoke(RequestReplyFunction.java:1
[jira] [Comment Edited] (FLINK-23890) CepOperator may create a large number of timers and cause performance problems
[
https://issues.apache.org/jira/browse/FLINK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513199#comment-17513199
]
Nicholas Jiang edited comment on FLINK-23890 at 3/28/22, 7:43 AM:
--
[~mayuehappy], IMO, when the watermark advances, the timers of all keys will be
triggered and removed from the eventTimeTimersQueue in TimerService. Therefore,
the number of the timers in TimerService will not be large. WDYT?
was (Author: nicholasjiang):
[~mayuehappy], after the offline discussion, the register of the EventTimeTimer
could be reduced by increasing the current watermark to EventTime+WindowTime,
not +1.
BTW, do you have interest to contribute a pull request to improve?
> CepOperator may create a large number of timers and cause performance problems
> --
>
> Key: FLINK-23890
> URL: https://issues.apache.org/jira/browse/FLINK-23890
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.12.1
>Reporter: Yue Ma
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-08-20-13-59-05-977.png
>
>
> There are two situations in the CepOperator that may register the time when
> dealing with EventTime.
> when the processElement will buffer the data first, and then register a timer
> with a timestamp of watermark+1.
> {code:java}
> if (timestamp > timerService.currentWatermark()) {
> // we have an event with a valid timestamp, so
> // we buffer it until we receive the proper watermark.
> saveRegisterWatermarkTimer();
> bufferEvent(value, timestamp);
> }{code}
> The other is when the EventTimer is triggered, if sortedTimestamps or
> partialMatches are not empty, a timer will also be registered.
> {code:java}
> if (!sortedTimestamps.isEmpty() || !partialMatches.isEmpty()) {
> saveRegisterWatermarkTimer();
> }{code}
>
> The problem is, if the partialMatches corresponding to each of my keys are
> not empty. Then every time the watermark advances, the timers of all keys
> will be triggered, and then a new EventTimer is re-registered under each key.
> When the number of task keys is very large, this operation greatly affects
> performance.
> !https://code.byted.org/inf/flink/uploads/91aee639553df07fa376cf2865e91fd2/image.png!
> I think it is unnecessary to register EventTimer frequently like this and can
> we make the following changes?
> When an event comes, the timestamp of the EventTimer we registered is equal
> to the EventTime of this event instead of watermark + 1.
> When a new ComputionState with window is created (like *withIn* pattern ),
> we use the timeout of this window to create EventTimer (EventTime +
> WindowTime).
> After making such an attempt in our test environment, the number of
> registered timers has been greatly reduced, and the performance has been
> greatly improved.
> !https://code.byted.org/inf/flink/uploads/24b85492c6a34a35c4445a4fd46c8363/image.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) * 302fcea19673a7e33216c2e8891038543c78723e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19208: [FLINK-26814][Deployment / YARN]It reported "should only have one jar" when submitting PyFlink jobs in YARN application mode
flinkbot edited a comment on pull request #19208: URL: https://github.com/apache/flink/pull/19208#issuecomment-1075903940 ## CI report: * 88b2acfe4ae9c93011ff273d00d3d0d9fb138af7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33804) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26862) Link the Github repository from Operator documentation
[ https://issues.apache.org/jira/browse/FLINK-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26862: --- Labels: pull-request-available (was: ) > Link the Github repository from Operator documentation > -- > > Key: FLINK-26862 > URL: https://issues.apache.org/jira/browse/FLINK-26862 > Project: Flink > Issue Type: Sub-task >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (FLINK-23890) CepOperator may create a large number of timers and cause performance problems
[ https://issues.apache.org/jira/browse/FLINK-23890 ]
Nicholas Jiang deleted comment on FLINK-23890:
was (Author: nicholasjiang):
[~mayuehappy], IMO, when the watermark advances, the timers of all keys will be
triggered and removed from the eventTimeTimersQueue in TimerService. Therefore,
the number of the timers in TimerService will not be large. WDYT?
> CepOperator may create a large number of timers and cause performance problems
> --
>
> Key: FLINK-23890
> URL: https://issues.apache.org/jira/browse/FLINK-23890
> Project: Flink
> Issue Type: Improvement
> Components: Library / CEP
>Affects Versions: 1.12.1
>Reporter: Yue Ma
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-08-20-13-59-05-977.png
>
>
> There are two situations in the CepOperator that may register the time when
> dealing with EventTime.
> when the processElement will buffer the data first, and then register a timer
> with a timestamp of watermark+1.
> {code:java}
> if (timestamp > timerService.currentWatermark()) {
> // we have an event with a valid timestamp, so
> // we buffer it until we receive the proper watermark.
> saveRegisterWatermarkTimer();
> bufferEvent(value, timestamp);
> }{code}
> The other is when the EventTimer is triggered, if sortedTimestamps or
> partialMatches are not empty, a timer will also be registered.
> {code:java}
> if (!sortedTimestamps.isEmpty() || !partialMatches.isEmpty()) {
> saveRegisterWatermarkTimer();
> }{code}
>
> The problem is, if the partialMatches corresponding to each of my keys are
> not empty. Then every time the watermark advances, the timers of all keys
> will be triggered, and then a new EventTimer is re-registered under each key.
> When the number of task keys is very large, this operation greatly affects
> performance.
> !https://code.byted.org/inf/flink/uploads/91aee639553df07fa376cf2865e91fd2/image.png!
> I think it is unnecessary to register EventTimer frequently like this and can
> we make the following changes?
> When an event comes, the timestamp of the EventTimer we registered is equal
> to the EventTime of this event instead of watermark + 1.
> When a new ComputionState with window is created (like *withIn* pattern ),
> we use the timeout of this window to create EventTimer (EventTime +
> WindowTime).
> After making such an attempt in our test environment, the number of
> registered timers has been greatly reduced, and the performance has been
> greatly improved.
> !https://code.byted.org/inf/flink/uploads/24b85492c6a34a35c4445a4fd46c8363/image.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration
[
https://issues.apache.org/jira/browse/FLINK-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513219#comment-17513219
]
Gyula Fora commented on FLINK-26881:
thanks [~wangyang0918] are you planning to work on this? would be great to
include this in the first RC today
> Stop with savepoint should pick up the targetDirectory from Flink
> configuration
> ---
>
> Key: FLINK-26881
> URL: https://issues.apache.org/jira/browse/FLINK-26881
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Priority: Blocker
> Fix For: kubernetes-operator-0.1.0
>
>
> Upgrading stateless FlinkDeployment to savepoint still could not work.
> Because {{FlinkService#cancelJob}} does not pick up the target directory in
> the flink configuration.
>
> {code:java}
> case SAVEPOINT:
> String savepoint =
> clusterClient
> .stopWithSavepoint(jobID, false, null)
> .get(1, TimeUnit.MINUTES); {code}
> We should use configured savepoint directory instead of {{null}} in the above
> implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-26873) Align the helm chart version with the flink operator
[ https://issues.apache.org/jira/browse/FLINK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Márton Balassi closed FLINK-26873. -- Resolution: Fixed Fixed via [1bcbbf8|https://github.com/apache/flink-kubernetes-operator/commit/1bcbbf8b6c1b19b19e6b662b679db3f95957d871] > Align the helm chart version with the flink operator > > > Key: FLINK-26873 > URL: https://issues.apache.org/jira/browse/FLINK-26873 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Aitozi >Assignee: Gyula Fora >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-0.1.0 > > > Now the flink-operator helm chart version is 1.0.13. I think it should be > aligned to the flink-operator version during release -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19221: [FLINK-26842][python] Remove scala-bridge dependency
flinkbot edited a comment on pull request #19221: URL: https://github.com/apache/flink/pull/19221#issuecomment-1077412156 ## CI report: * 14af40576119ba874cc3288111c9a1dba64123e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33746) * 25af9a0b752dbfc23780d651ed7caf3471abb526 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19221: [FLINK-26842][python] Remove scala-bridge dependency
flinkbot edited a comment on pull request #19221: URL: https://github.com/apache/flink/pull/19221#issuecomment-1077412156 ## CI report: * 14af40576119ba874cc3288111c9a1dba64123e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33746) * 25af9a0b752dbfc23780d651ed7caf3471abb526 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33807) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26214) Publish Kubernetes operator to container registry
[ https://issues.apache.org/jira/browse/FLINK-26214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26214: --- Labels: pull-request-available (was: ) > Publish Kubernetes operator to container registry > - > > Key: FLINK-26214 > URL: https://issues.apache.org/jira/browse/FLINK-26214 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Zili Chen >Assignee: Márton Balassi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-0.1.0 > > > Created from > https://github.com/apache/flink-kubernetes-operator/pull/4#issuecomment-1042717476. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26617) Pass Kafka headers to remote functions and egresses
[ https://issues.apache.org/jira/browse/FLINK-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513227#comment-17513227 ] Till Rohrmann commented on FLINK-26617: --- Thanks for creating this ticket [~Fil Karnicki]. I think the topic of tracing is very interesting as it opens SF up for better understanding what is going on. Ideally, we can solve this problem generically as tracing is also interesting for other ingresses and SF calls in general. I think what we would need to add to the runtime is support for creating/deriving spans and propagating their contexts. I am not sure whether this information really needs to be forwarded to a remote function unless the remote function can spawn external calls that need the context as well. That way we don't need SDK specific handling logic. However, we would need to adjust the ingresses and egresses to understand the span/context field of a message. > Pass Kafka headers to remote functions and egresses > --- > > Key: FLINK-26617 > URL: https://issues.apache.org/jira/browse/FLINK-26617 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Fil Karnicki >Priority: Minor > > Typically OpenTelemetry (FLINK-22390) tracing spans get passed in kafka > headers. We could be passing not only the Kafka ConsumerRecord value, but > also the headers to remote functions, if the user configures their kafka > ingress to do so > Similarly, kafka egresses could be configurable so that headers get passed on > via the KafkaProducerRecord proto to kafka -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19225: [FLINK-26799][Runtime/StateBackends] fix StateChangeFormat#read and e…
flinkbot edited a comment on pull request #19225: URL: https://github.com/apache/flink/pull/19225#issuecomment-1077559112 ## CI report: * 5b98d0047303aedcb0676c34b3ac8aca98c148e9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33793) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #19135: [FLINK-26708] TimestampsAndWatermarksOperator should not propagate WatermarkStatus
dawidwys merged pull request #19135: URL: https://github.com/apache/flink/pull/19135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26628) Use REST program arguments in StatefulFunctionsJob
[
https://issues.apache.org/jira/browse/FLINK-26628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513229#comment-17513229
]
Till Rohrmann commented on FLINK-26628:
---
I think [~igal] is right here [~Fil Karnicki]. Flink's web submission has some
limitations and this is one of them if I am not mistaken. In general, the
recommendation in the Flink community is to not use the web submission but
rather to use the cli. Hence, I believe that this issue is a Flink and not a SF
issue.
> Use REST program arguments in StatefulFunctionsJob
> --
>
> Key: FLINK-26628
> URL: https://issues.apache.org/jira/browse/FLINK-26628
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
>Reporter: Fil Karnicki
>Priority: Major
>
> Currently the Program Arguments passed into the REST api don't get used by
> StreamingExecutionEnvironment in StatefulFunctionsJob (in this case the
> checkpointing will not be set)
> {code:java}
> --execution.checkpointing.interval 1000 --state.backend rocksdb
> --state.checkpoint-storage filesystem --state.checkpoints.dir file:///tmp/
> --statefun.embedded true {code}
> Conversely, Flink CLI params *do* get used by the
> StreamingExecutionEnvironment in statefun jobs
> {code:java}
> flink run -Dexecution.checkpointing.interval=1000 -Dstate.backend=rocksdb
> -Dstate.checkpoint-storage=filesystem -Dstate.checkpoints.dir=file:///tmp/
> -Dstatefun.embedded=true myjar.jar{code}
>
> To reproduce,
> # clone and run mvn package on
> [https://github.com/FilKarnicki/statefun-flinkjob/tree/argsNotUsedViaRest]
> # run docker-compose up in flinkjob/docker-compose
> # observe checkpointing happening for this job
> # go to [http://localhost:8081/#/submit] and submit
> flinkjob-1.0-SNAPSHOT.jar again manually from target with program arguments
> {code:java}
> --execution.checkpointing.interval 1000 --state.backend rocksdb
> --state.checkpoint-storage filesystem --state.checkpoints.dir file:///tmp/
> --statefun.embedded true {code}
> 5. observe no checkpointing happening for the second job
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zoltar9264 commented on pull request #19225: [FLINK-26799][Runtime/StateBackends] fix StateChangeFormat#read and e…
zoltar9264 commented on pull request #19225: URL: https://github.com/apache/flink/pull/19225#issuecomment-1080339676 Hi @rkhachatryan , I rebased it to latest master branch , then the Ci passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-26708) TimestampsAndWatermarksOperator should not propagate WatermarkStatus
[ https://issues.apache.org/jira/browse/FLINK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-26708. Fix Version/s: 1.15.0 1.13.7 1.14.5 Resolution: Fixed Fixed in: * master ** d87808fe8a2fe6538b902056490395ae8597a48b * 1.15.0 ** f858421d67a27c155accdd7da9117c29541f8939 * 1.14.5 ** df0363de3e5bbd284046600bf713df17741a39f9 * 1.13.7 ** 1e723c568a76d6b37b08404575878a320f5c7f82 > TimestampsAndWatermarksOperator should not propagate WatermarkStatus > > > Key: FLINK-26708 > URL: https://issues.apache.org/jira/browse/FLINK-26708 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.15.0, 1.14.4 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.13.7, 1.14.5 > > > The lifecycle/scope of WatermarkStatus is tightly coupled with watermarks. > Upstream watermarks are cut off in the TimestampsAndWatermarksOperator and > therefore watermark statuses should be cut off as well. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] JingsongLi merged pull request #19072: [FLINK-11388][fs] Add Aliyun OSS recoverable writer
JingsongLi merged pull request #19072: URL: https://github.com/apache/flink/pull/19072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-11388) Add an OSS RecoverableWriter
[ https://issues.apache.org/jira/browse/FLINK-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-11388. Resolution: Fixed master: 672d26e831047a99b4d6432fe0046ff036279887 > Add an OSS RecoverableWriter > > > Key: FLINK-11388 > URL: https://issues.apache.org/jira/browse/FLINK-11388 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem, FileSystems >Affects Versions: 1.7.1 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > Time Spent: 10m > Remaining Estimate: 0h > > OSS offers persistence only after uploads or multi-part uploads complete. In > order to make streaming uses OSS as sink, we should implement a Recoverable > writer. This writer will snapshot and store multi-part upload information and > recover from those information when failure occurs -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration
[
https://issues.apache.org/jira/browse/FLINK-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513236#comment-17513236
]
Yang Wang commented on FLINK-26881:
---
I will attach a PR soon.
> Stop with savepoint should pick up the targetDirectory from Flink
> configuration
> ---
>
> Key: FLINK-26881
> URL: https://issues.apache.org/jira/browse/FLINK-26881
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Priority: Blocker
> Fix For: kubernetes-operator-0.1.0
>
>
> Upgrading stateless FlinkDeployment to savepoint still could not work.
> Because {{FlinkService#cancelJob}} does not pick up the target directory in
> the flink configuration.
>
> {code:java}
> case SAVEPOINT:
> String savepoint =
> clusterClient
> .stopWithSavepoint(jobID, false, null)
> .get(1, TimeUnit.MINUTES); {code}
> We should use configured savepoint directory instead of {{null}} in the above
> implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration
[
https://issues.apache.org/jira/browse/FLINK-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang reassigned FLINK-26881:
-
Assignee: Yang Wang
> Stop with savepoint should pick up the targetDirectory from Flink
> configuration
> ---
>
> Key: FLINK-26881
> URL: https://issues.apache.org/jira/browse/FLINK-26881
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Blocker
> Fix For: kubernetes-operator-0.1.0
>
>
> Upgrading stateless FlinkDeployment to savepoint still could not work.
> Because {{FlinkService#cancelJob}} does not pick up the target directory in
> the flink configuration.
>
> {code:java}
> case SAVEPOINT:
> String savepoint =
> clusterClient
> .stopWithSavepoint(jobID, false, null)
> .get(1, TimeUnit.MINUTES); {code}
> We should use configured savepoint directory instead of {{null}} in the above
> implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ZhangChaoming commented on a change in pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
ZhangChaoming commented on a change in pull request #18386:
URL: https://github.com/apache/flink/pull/18386#discussion_r836179237
##
File path:
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/dql/SqlShowDatabases.java
##
@@ -49,8 +70,44 @@ public SqlOperator getOperator() {
return Collections.EMPTY_LIST;
}
+public String getCatalogName() {
+return Objects.isNull(this.catalogName) ? null :
catalogName.getSimple();
+}
+
+public boolean isNotLike() {
+return notLike;
+}
+
+public String getPreposition() {
+return preposition;
+}
+
+public String getLikeSqlPattern() {
+return Objects.isNull(this.likeLiteral) ? null :
likeLiteral.getValueAs(String.class);
+}
+
+public SqlCharStringLiteral getLikeLiteral() {
+return likeLiteral;
+}
+
+public boolean isWithLike() {
+return Objects.nonNull(likeLiteral);
+}
+
@Override
public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
-writer.keyword("SHOW DATABASES");
+if (this.preposition == null) {
+writer.keyword("SHOW DATABASES");
+} else if (catalogName != null) {
+writer.keyword("SHOW DATABASES " + this.preposition);
+catalogName.unparse(writer, leftPrec, rightPrec);
+}
+if (likeLiteral != null) {
+if (notLike) {
+writer.keyword(String.format("NOT LIKE '%s'",
getLikeSqlPattern()));
Review comment:
Do you think the result of `show databases like ;` is out of our exepect
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26862) Link the Github repository from Operator documentation
[ https://issues.apache.org/jira/browse/FLINK-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26862: --- Component/s: Kubernetes Operator > Link the Github repository from Operator documentation > -- > > Key: FLINK-26862 > URL: https://issues.apache.org/jira/browse/FLINK-26862 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26862) Link the Github repository from Operator documentation
[ https://issues.apache.org/jira/browse/FLINK-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-26862: --- Fix Version/s: kubernetes-operator-0.1.0 > Link the Github repository from Operator documentation > -- > > Key: FLINK-26862 > URL: https://issues.apache.org/jira/browse/FLINK-26862 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-0.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-26862) Link the Github repository from Operator documentation
[ https://issues.apache.org/jira/browse/FLINK-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-26862. -- Resolution: Fixed fixed via 33eb09b749cc57b4badfcfb4533820b4e5454c23 > Link the Github repository from Operator documentation > -- > > Key: FLINK-26862 > URL: https://issues.apache.org/jira/browse/FLINK-26862 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-0.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-22008) writing metadata is not an atomic operation, we should add a commit logic
[ https://issues.apache.org/jira/browse/FLINK-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-22008. -- Resolution: Duplicate > writing metadata is not an atomic operation, we should add a commit logic > - > > Key: FLINK-22008 > URL: https://issues.apache.org/jira/browse/FLINK-22008 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.12.2 >Reporter: xiaogang zhou >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > > writing metadata is not an atomic operation, some logic can cause there is a > metadata file in the checkpoint dir, but the data is corrupted if the > jobmanager crash while writing the metadata. > > So we should consider to add commit operation in the checkpoint storage > location -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26789) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
[
https://issues.apache.org/jira/browse/FLINK-26789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513242#comment-17513242
]
Yangze Guo commented on FLINK-26789:
Can we ignore this test atm as it is quite unstable?
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
> --
>
> Key: FLINK-26789
> URL: https://issues.apache.org/jira/browse/FLINK-26789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> [This
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=894&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1&l=5687]
> failed due to
> {{RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState}}:
> {code}
> Mar 21 17:05:32 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState
> Time elapsed: 23.966 s <<< FAILURE!
> Mar 21 17:05:32 java.lang.AssertionError: expected:<[(0,24000), (1,22500),
> (0,34500), (1,33000), (0,21000), (0,45000), (2,31500), (2,42000), (1,6000),
> (0,28500), (0,52500), (2,15000), (1,3000), (1,51000), (0,1500), (0,49500),
> (2,12000), (2,6), (0,36000), (1,10500), (1,58500), (0,46500), (0,9000),
> (0,57000), (2,19500), (2,43500), (1,7500), (1,55500), (2,3), (1,18000),
> (0,54000), (2,40500), (1,4500), (0,16500), (2,27000), (1,39000), (2,13500),
> (1,25500), (0,37500), (0,61500), (2,0), (2,48000)]> but was:<[(1,22500),
> (1,33000), (0,21000), (2,18000), (1,6000), (0,20500), (0,52500), (0,15000),
> (0,31000), (2,12000), (2,6), (0,36000), (1,58500), (1,10500), (0,46500),
> (0,25000), (0,41000), (0,9000), (0,57000), (2,43500), (0,3), (1,4500),
> (2,27000), (1,15000), (0,35000), (0,19000), (0,3000), (1,25500), (0,61500),
> (2,48000), (2,0), (0,24000), (0,34500), (0,45000), (2,31500), (1,19500),
> (2,1), (2,42000), (0,12500), (0,28500), (2,15000), (1,3000), (1,51000),
> (0,23000), (0,49500), (0,1500), (0,33000), (0,1000), (2,19500), (1,7500),
> (1,55500), (2,3), (1,18000), (0,6000), (0,38000), (0,54000), (2,40500),
> (0,500), (0,16500), (1,39000), (1,7000), (0,11000), (2,13500), (0,37500)]>
> Mar 21 17:05:32 at org.junit.Assert.fail(Assert.java:89)
> Mar 21 17:05:32 at org.junit.Assert.failNotEquals(Assert.java:835)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:120)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:146)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:218)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:122)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState(RescaleCheckpointManuallyITCase.java:88)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] YuriGusev commented on a change in pull request #18518: [FLINK-24229][connectors/dynamodb] Added DynamoDB connector
YuriGusev commented on a change in pull request #18518:
URL: https://github.com/apache/flink/pull/18518#discussion_r836187182
##
File path:
flink-connectors/flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java
##
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import
org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import
org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More
details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More
details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * The {@link DynamoDbAsyncClient} used here may be configured in the
standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code
AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter extends AsyncSinkWriter {
+private static final Logger LOG =
LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+/* A counter for the total number of records that have encountered an
error during put */
+private final Counter numRecordsOutErrorsCounter;
+
+/* The sink writer metric group */
+private final SinkWriterMetricGroup metrics;
+
+private final DynamoDbTablesConfig tablesConfig;
+private final DynamoDbAsyncClient client;
+private final boolean failOnError;
+
+public DynamoDbSinkWriter(
+ElementConverter elementConverter,
+Sink.InitContext context,
+int maxBatchSize,
+int maxInFlightRequests,
+int maxBufferedRequests,
+long maxBatchSizeInBytes,
+long maxTimeInBufferMS,
+long maxRecordSizeInBytes,
+boolean failOnError,
+DynamoDbTablesConfig tablesConfig,
+Properties dynamoDbClientProperties) {
+super(
+elementConverter,
+context,
+maxBatchSize,
+maxInFlightRequests,
+maxBufferedRequests,
+maxBatchSizeInBytes,
+maxTimeInBufferMS,
+maxRecordSizeInBytes);
+this.failOnError = failOnError;
+this.tablesConfig = tablesConfig;
+this.metrics = context.metricGroup();
+this.numRecordsOutErrorsCounter =
metrics.getNumRecordsOutErrorsCounter();
+this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+}
+
+@Override
+protected void submitRequestEntries(
+List requestEntries,
+Consumer> requestResultConsumer) {
+
+TableRequestsContainer container = new
TableRequestsContainer(tablesConfig);
+requestEntries.forEach(container::put);
Review comment:
yes good point, this is the same how boto3 behaves. May be we should
rename batch size property to MAX batch size
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-26760:
---
Priority: Major (was: Critical)
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: PerformanceTest.java, example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSy
[GitHub] [flink] flinkbot edited a comment on pull request #18345: [FLINK-25440][doc][pulsar] Both stopCursor and startCursor now uses publish time instead of event time
flinkbot edited a comment on pull request #18345: URL: https://github.com/apache/flink/pull/18345#issuecomment-1011821419 ## CI report: * 9d6f199f94a022807e2252c805bac9f8256c7924 UNKNOWN * 97e8a2f234adc5ea1888b43e2d55412902a5221b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33798) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19023: [FLINK-25705][docs]Translate "Metric Reporters" page of "Deployment" …
flinkbot edited a comment on pull request #19023: URL: https://github.com/apache/flink/pull/19023#issuecomment-1062931101 ## CI report: * d3ec7878c8779c901bec0a7497b247cbe354b96e UNKNOWN * 65c05c107320acecd49d3264212cd34d1b75beb2 UNKNOWN * e36f4139caaa7aed67336e59fe56634b3297aa0f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33673) * 3289c9b3177cad0bfbade3f12044b4610513a9a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33812) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26790) Description image about Unaligned Checkpointing maybe incorrect.
[
https://issues.apache.org/jira/browse/FLINK-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513247#comment-17513247
]
Piotr Nowojski commented on FLINK-26790:
I think [~assassinj]'s confusion is about the picture. On the first figure
captured "on first barrier", the checkpoint barrier for the lower connection is
in the middle of the output buffers (between buffers "e" and "d"), which
contradicts what is described below:
{quote}
by adding it to the end of the output buffers.
{quote}
I think indeed the graphic should be changed somehow.
I think I understand the [~arvid]'s intention behind it, that it was supposed
to depict a scenario where two checkpoint barriers from two different upstream
operators are arriving at different time, and the second channel is keep
persisting the in-flight data until it receives it's checkpoint barrier, but
indeed that's misleading as it is. Especially that it shows two operators
connected to each other via two channels? Which apart of some esoteric scenario
with for example union-ing output and side output of the same operator is
impossible to achieve.
I would either simplify this graph, and focus on single input/output channel
case (parallelism = 1), or keep it with upstream parallelism = 2, but then:
# Clearly adding one more upstream operator
# Adding one more extra step, where in the first figure only first upstream
operator inject checkpoint barrier, and in the next figure second upstream
operator is doing it
WDYT?
> Description image about Unaligned Checkpointing maybe incorrect.
>
>
> Key: FLINK-26790
> URL: https://issues.apache.org/jira/browse/FLINK-26790
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.13.6, 1.14.4
>Reporter: jinghaihang
>Priority: Minor
> Attachments: image-2022-03-22-11-12-20-025.png
>
>
> Due to the second description ：
> * It immediately forwards the barrier to the downstream operator by adding
> it to the end of the output buffers.
> so i think the picture maybe misleading readers that barrier will not appear
> in the middle of the data stream.
>
> !image-2022-03-22-11-12-20-025.png|width=599,height=303!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26882) Unaligned checkpoint with 0s timeout would fail RescaleCheckpointManuallyITCase
Yun Tang created FLINK-26882:
Summary: Unaligned checkpoint with 0s timeout would fail
RescaleCheckpointManuallyITCase
Key: FLINK-26882
URL: https://issues.apache.org/jira/browse/FLINK-26882
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Tests
Reporter: Yun Tang
Fix For: 1.16.0
Once we make {{execution.checkpointing.unaligned: true}} and
{{execution.checkpointing.alignment-timeout: PT0S}}, the
RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState would fail
then.
Borken instances:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33776&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5623
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5626
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5629
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=5733
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a549b384-c55a-52c0-c451-00e0477ab6db&t=eef5922c-08d9-5ba3-7299-8393476594e7&l=12575
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=5838
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=b0a398c0-685b-599c-eb57-c8c2a771138e&t=747432ad-a576-5911-1e2a-68c6bedc248a&l=12931
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=5682
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] ZhangChaoming commented on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector
ZhangChaoming commented on pull request #18151: URL: https://github.com/apache/flink/pull/18151#issuecomment-1080374140 @MartijnVisser I think my code is safe, since the error was caused by testing `test_ci finegrained_resource_management`. And This branch has no conflict with the base branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26789) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
[
https://issues.apache.org/jira/browse/FLINK-26789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513249#comment-17513249
]
Yun Tang commented on FLINK-26789:
--
I have created another ticket https://issues.apache.org/jira/browse/FLINK-26789
to focus this new problem. Let's discuss there.
cc [~guoyangze], [~roman], [~gaoyunhaii], [~pnowojski]
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
> --
>
> Key: FLINK-26789
> URL: https://issues.apache.org/jira/browse/FLINK-26789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> [This
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=894&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1&l=5687]
> failed due to
> {{RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState}}:
> {code}
> Mar 21 17:05:32 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState
> Time elapsed: 23.966 s <<< FAILURE!
> Mar 21 17:05:32 java.lang.AssertionError: expected:<[(0,24000), (1,22500),
> (0,34500), (1,33000), (0,21000), (0,45000), (2,31500), (2,42000), (1,6000),
> (0,28500), (0,52500), (2,15000), (1,3000), (1,51000), (0,1500), (0,49500),
> (2,12000), (2,6), (0,36000), (1,10500), (1,58500), (0,46500), (0,9000),
> (0,57000), (2,19500), (2,43500), (1,7500), (1,55500), (2,3), (1,18000),
> (0,54000), (2,40500), (1,4500), (0,16500), (2,27000), (1,39000), (2,13500),
> (1,25500), (0,37500), (0,61500), (2,0), (2,48000)]> but was:<[(1,22500),
> (1,33000), (0,21000), (2,18000), (1,6000), (0,20500), (0,52500), (0,15000),
> (0,31000), (2,12000), (2,6), (0,36000), (1,58500), (1,10500), (0,46500),
> (0,25000), (0,41000), (0,9000), (0,57000), (2,43500), (0,3), (1,4500),
> (2,27000), (1,15000), (0,35000), (0,19000), (0,3000), (1,25500), (0,61500),
> (2,48000), (2,0), (0,24000), (0,34500), (0,45000), (2,31500), (1,19500),
> (2,1), (2,42000), (0,12500), (0,28500), (2,15000), (1,3000), (1,51000),
> (0,23000), (0,49500), (0,1500), (0,33000), (0,1000), (2,19500), (1,7500),
> (1,55500), (2,3), (1,18000), (0,6000), (0,38000), (0,54000), (2,40500),
> (0,500), (0,16500), (1,39000), (1,7000), (0,11000), (2,13500), (0,37500)]>
> Mar 21 17:05:32 at org.junit.Assert.fail(Assert.java:89)
> Mar 21 17:05:32 at org.junit.Assert.failNotEquals(Assert.java:835)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:120)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:146)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:218)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:122)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState(RescaleCheckpointManuallyITCase.java:88)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-26789) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
[
https://issues.apache.org/jira/browse/FLINK-26789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang closed FLINK-26789.
Resolution: Fixed
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
> --
>
> Key: FLINK-26789
> URL: https://issues.apache.org/jira/browse/FLINK-26789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> [This
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=894&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1&l=5687]
> failed due to
> {{RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState}}:
> {code}
> Mar 21 17:05:32 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState
> Time elapsed: 23.966 s <<< FAILURE!
> Mar 21 17:05:32 java.lang.AssertionError: expected:<[(0,24000), (1,22500),
> (0,34500), (1,33000), (0,21000), (0,45000), (2,31500), (2,42000), (1,6000),
> (0,28500), (0,52500), (2,15000), (1,3000), (1,51000), (0,1500), (0,49500),
> (2,12000), (2,6), (0,36000), (1,10500), (1,58500), (0,46500), (0,9000),
> (0,57000), (2,19500), (2,43500), (1,7500), (1,55500), (2,3), (1,18000),
> (0,54000), (2,40500), (1,4500), (0,16500), (2,27000), (1,39000), (2,13500),
> (1,25500), (0,37500), (0,61500), (2,0), (2,48000)]> but was:<[(1,22500),
> (1,33000), (0,21000), (2,18000), (1,6000), (0,20500), (0,52500), (0,15000),
> (0,31000), (2,12000), (2,6), (0,36000), (1,58500), (1,10500), (0,46500),
> (0,25000), (0,41000), (0,9000), (0,57000), (2,43500), (0,3), (1,4500),
> (2,27000), (1,15000), (0,35000), (0,19000), (0,3000), (1,25500), (0,61500),
> (2,48000), (2,0), (0,24000), (0,34500), (0,45000), (2,31500), (1,19500),
> (2,1), (2,42000), (0,12500), (0,28500), (2,15000), (1,3000), (1,51000),
> (0,23000), (0,49500), (0,1500), (0,33000), (0,1000), (2,19500), (1,7500),
> (1,55500), (2,3), (1,18000), (0,6000), (0,38000), (0,54000), (2,40500),
> (0,500), (0,16500), (1,39000), (1,7000), (0,11000), (2,13500), (0,37500)]>
> Mar 21 17:05:32 at org.junit.Assert.fail(Assert.java:89)
> Mar 21 17:05:32 at org.junit.Assert.failNotEquals(Assert.java:835)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:120)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:146)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:218)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:122)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState(RescaleCheckpointManuallyITCase.java:88)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] imaffe commented on pull request #18345: [FLINK-25440][doc][pulsar] Both stopCursor and startCursor now uses publish time instead of event time
imaffe commented on pull request #18345: URL: https://github.com/apache/flink/pull/18345#issuecomment-1080374813 @MartijnVisser Hi Martin, are you feeling better ? I'll rebase and resolve the conflicts once you finished review ~ Thanks ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26882) Unaligned checkpoint with 0s timeout would fail RescaleCheckpointManuallyITCase
[
https://issues.apache.org/jira/browse/FLINK-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang updated FLINK-26882:
-
Priority: Critical (was: Major)
> Unaligned checkpoint with 0s timeout would fail
> RescaleCheckpointManuallyITCase
> ---
>
> Key: FLINK-26882
> URL: https://issues.apache.org/jira/browse/FLINK-26882
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Reporter: Yun Tang
>Priority: Critical
> Fix For: 1.16.0
>
>
> Once we make {{execution.checkpointing.unaligned: true}} and
> {{execution.checkpointing.alignment-timeout: PT0S}}, the
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState would
> fail then.
> Borken instances:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33776&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5623
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5626
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5629
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=5733
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a549b384-c55a-52c0-c451-00e0477ab6db&t=eef5922c-08d9-5ba3-7299-8393476594e7&l=12575
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=5838
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=b0a398c0-685b-599c-eb57-c8c2a771138e&t=747432ad-a576-5911-1e2a-68c6bedc248a&l=12931
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=5682
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18967: [FLINK-26460][table-planner]Fix Unsupported type when convertTypeToSpec: MAP
flinkbot edited a comment on pull request #18967: URL: https://github.com/apache/flink/pull/18967#issuecomment-1057833136 ## CI report: * ac4a22da8f2a89736cca5baea43382056f874d5a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33800) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19235: [FLINK-26848][JDBC]write data when disable flush-interval and max-rows
flinkbot edited a comment on pull request #19235: URL: https://github.com/apache/flink/pull/19235#issuecomment-1077732821 ## CI report: * c071084ab31d8819b54be2afc701a9517b70f58f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33801) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26762) Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being blocked
[ https://issues.apache.org/jira/browse/FLINK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513250#comment-17513250 ] fanrui commented on FLINK-26762: Hi [~pnowojski] [~arvid] , could you help take a look in your free time? Thanks a lot. > Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being > blocked > --- > > Key: FLINK-26762 > URL: https://issues.apache.org/jira/browse/FLINK-26762 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Network >Affects Versions: 1.13.0, 1.14.0, 1.15.0 >Reporter: fanrui >Priority: Major > Fix For: 1.16.0 > > > In some past JIRAs of Unaligned Checkpoint, the community has added the > recordWriter.isAvaliable() to reduce block for single record write. But for > large record, flatmap or broadcast watermark, they may need more buffer. > Can we add the overdraft buffer in BufferPool to reduce unaligned checkpoint > being blocked? > h2. Overdraft Buffer mechanism > Add the configuration of > 'taskmanager.network.memory.overdraft-buffers-per-gate=5'. > When requestMemory is called and the bufferPool is insufficient, the > bufferPool will allow the Task to overdraw up to 5 MemorySegments. And > bufferPool will be unavailable until all overdrawn buffers are consumed by > downstream tasks. Then the task will wait for bufferPool being available. > From the above, we have the following benefits: > * For scenarios that require multiple buffers, the Task releases the > Checkpoint lock, so the Unaligned Checkpoint can be completed quickly. > * We can control the memory usage to prevent memory leak. > * It just needs a litter memory, and can improve the stability of the Task > under back pressure. > * Users can increase the overdraft-buffers to adapt the scenarios that > require more buffers. > > Masters, please correct me if I'm wrong, thanks a lot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] pnowojski commented on a change in pull request #19198: [FLINK-26783] Do not trigger global failover if failed during commiting side-effects during stop-with-savepoint
pnowojski commented on a change in pull request #19198:
URL: https://github.com/apache/flink/pull/19198#discussion_r836203434
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/stopwithsavepoint/StopWithSavepointTerminationHandlerImpl.java
##
@@ -167,16 +166,13 @@ private void
handleAnyExecutionNotFinished(Set notFinishedExecut
*/
private void terminateExceptionallyWithGlobalFailover(
Iterable unfinishedExecutionStates, String
savepointPath) {
-String errorMessage =
-String.format(
-"Inconsistent execution state after stopping with
savepoint. At least one execution is still in one of the following states: %s.
A global fail-over is triggered to recover the job %s.",
-StringUtils.join(unfinishedExecutionStates, ", "),
jobId);
-FlinkException inconsistentFinalStateException = new
FlinkException(errorMessage);
+StopWithSavepointException inconsistentFinalStateException =
+new StopWithSavepointException(savepointPath, jobId);
log.warn(
-"A savepoint was created at {} but the corresponding job {}
didn't terminate successfully.",
-savepointPath,
-jobId,
+"Inconsistent execution state after stopping with savepoint.
At least one"
++ " execution is still in one of the following states:
{}.",
+StringUtils.join(unfinishedExecutionStates, ", "),
inconsistentFinalStateException);
scheduler.handleGlobalFailure(inconsistentFinalStateException);
Review comment:
Shouldn't we change something around the failover behaviour?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/stopwithsavepoint/StopWithSavepointException.java
##
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.stopwithsavepoint;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.throwable.ThrowableAnnotation;
+import org.apache.flink.runtime.throwable.ThrowableType;
+import org.apache.flink.util.FlinkException;
+
+/**
+ * Exception thrown when a savepoint has been created successfully when
stopping with savepoint, but
+ * the job has not finished. In that case side-effects might have not been
committed. This exception
+ * is used to communicate that to the use.
+ */
+@Experimental
+@ThrowableAnnotation(ThrowableType.NonRecoverableError)
+public class StopWithSavepointException extends FlinkException {
Review comment:
`StopWithSavepointStoppingException`?
`StopWithSavepointExceptionWhenStopping`?
Otherwise, as it is, someone might just (mis)use
`StopWithSavepointException` in the future to indicate any type of exception
during the stop-with-savepoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-26803) Merge small ChannelState file for Unaligned Checkpoint
[ https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513251#comment-17513251 ] fanrui commented on FLINK-26803: Hi [~pnowojski] [~arvid] , could you help take a look in your free time? Thanks a lot. > Merge small ChannelState file for Unaligned Checkpoint > -- > > Key: FLINK-26803 > URL: https://issues.apache.org/jira/browse/FLINK-26803 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Network >Reporter: fanrui >Priority: Major > > When making an unaligned checkpoint, the number of ChannelState files is > TaskNumber * subtaskNumber. For high parallelism job, it writes too many > small files. It causes high load for hdfs NN. > > In our production, a job writes more than 50K small files for each Unaligned > Checkpoint. Could we merge these files before write FileSystem? We can > configure the maximum number of files each TM can write in a single Unaligned > Checkpoint. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] ZhangChaoming edited a comment on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector
ZhangChaoming edited a comment on pull request #18151: URL: https://github.com/apache/flink/pull/18151#issuecomment-1080374140 @MartijnVisser I think my code is safe, since the error was caused by testing `test_ci finegrained_resource_management`. And This branch has no conflict with the base branch. Ref [FLINK-26882](https://issues.apache.org/jira/browse/FLINK-26882) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #19198: [FLINK-26783] Do not trigger global failover if failed during commiting side-effects during stop-with-savepoint
dawidwys commented on a change in pull request #19198:
URL: https://github.com/apache/flink/pull/19198#discussion_r836205477
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/stopwithsavepoint/StopWithSavepointTerminationHandlerImpl.java
##
@@ -167,16 +166,13 @@ private void
handleAnyExecutionNotFinished(Set notFinishedExecut
*/
private void terminateExceptionallyWithGlobalFailover(
Iterable unfinishedExecutionStates, String
savepointPath) {
-String errorMessage =
-String.format(
-"Inconsistent execution state after stopping with
savepoint. At least one execution is still in one of the following states: %s.
A global fail-over is triggered to recover the job %s.",
-StringUtils.join(unfinishedExecutionStates, ", "),
jobId);
-FlinkException inconsistentFinalStateException = new
FlinkException(errorMessage);
+StopWithSavepointException inconsistentFinalStateException =
+new StopWithSavepointException(savepointPath, jobId);
log.warn(
-"A savepoint was created at {} but the corresponding job {}
didn't terminate successfully.",
-savepointPath,
-jobId,
+"Inconsistent execution state after stopping with savepoint.
At least one"
++ " execution is still in one of the following states:
{}.",
+StringUtils.join(unfinishedExecutionStates, ", "),
inconsistentFinalStateException);
scheduler.handleGlobalFailure(inconsistentFinalStateException);
Review comment:
It's done with the annotation on the exception:
`@ThrowableAnnotation(ThrowableType.NonRecoverableError)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-26738) Default value of StateDescriptor is valid when enable state ttl config
[
https://issues.apache.org/jira/browse/FLINK-26738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513252#comment-17513252
]
Jianhui Dong commented on FLINK-26738:
--
[~yunta], resolved conflict, please check it agian.
> Default value of StateDescriptor is valid when enable state ttl config
> --
>
> Key: FLINK-26738
> URL: https://issues.apache.org/jira/browse/FLINK-26738
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.15.0
>Reporter: Jianhui Dong
>Assignee: Jianhui Dong
>Priority: Major
> Labels: pull-request-available
>
> Suppose we declare a ValueState like following:
> {code:java}
> ValueStateDescriptor> descriptor =
> new ValueStateDescriptor<>(
> "average", // the state name
> TypeInformation.of(new TypeHint>()
> {}),
> Tuple2.of(0L, 0L));
> {code}
> and then we add state ttl config to the state:
> {code:java}
> descriptor.enableTimeToLive(StateTtlConfigUtil.createTtlConfig(6));
> {code}
> the default value Tuple2.of(0L, 0L) will be invalid and may cause NPE.
> I don't know if this is a bug cause I see @Deprecated in the comment of the
> ValueStateDescriptor constructor with argument defaultValue:
> {code:java}
> Use {@link #ValueStateDescriptor(String, TypeSerializer)} instead and manually
> * manage the default value by checking whether the contents of the
> state is {@code null}.
> {code}
> and if we decide not to use the defaultValue field in the class
> StateDescriptor, should we add @Deprecated annotation to the field
> defaultValue?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] liuzhuang2017 commented on a change in pull request #19248: Update JobLeaderIdService.java
liuzhuang2017 commented on a change in pull request #19248:
URL: https://github.com/apache/flink/pull/19248#discussion_r836207557
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/JobLeaderIdService.java
##
@@ -33,7 +33,7 @@
public interface JobLeaderIdService {
/**
- * Start the service with the given job leader actions.
+ * Start the service with the given job leader id actions.
Review comment:
@MartijnVisser Thank you for your reply, I will read this official
document carefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Tartarus0zm commented on pull request #18967: [FLINK-26460][table-planner]Fix Unsupported type when convertTypeToSpec: MAP
Tartarus0zm commented on pull request #18967: URL: https://github.com/apache/flink/pull/18967#issuecomment-1080387533 CI failure doesn't seem to be related to this patch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] luoyuxia opened a new pull request #19253: [hive] support nvl(null, null)
luoyuxia opened a new pull request #19253: URL: https://github.com/apache/flink/pull/19253 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19249: [FLINK-26092][table-runtime] Fix `JSON_OBJECTAGG` when emitting `NULL`
flinkbot edited a comment on pull request #19249: URL: https://github.com/apache/flink/pull/19249#issuecomment-1079884730 ## CI report: * 42c8b7d3793c949b54aba0899089d745721a0f34 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33783) * 46c05b37929a49451a6a7c029ba75366b30b7aef UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #18151: [FLINK-25379][connectors] Support limit push down in DATAGEN connector
MartijnVisser commented on pull request #18151: URL: https://github.com/apache/flink/pull/18151#issuecomment-1080391007 @ZhangChaoming Still, I wouldn't merge it unless CI has fully passed. Note: you'll need a review from probably @slinkydeveloper before this could be merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19249: [FLINK-26092][table-runtime] Fix `JSON_OBJECTAGG` when emitting `NULL`
flinkbot edited a comment on pull request #19249: URL: https://github.com/apache/flink/pull/19249#issuecomment-1079884730 ## CI report: * 42c8b7d3793c949b54aba0899089d745721a0f34 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33783) * 46c05b37929a49451a6a7c029ba75366b30b7aef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33813) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan merged pull request #19225: [FLINK-26799][Runtime/StateBackends] fix StateChangeFormat#read and e…
rkhachatryan merged pull request #19225: URL: https://github.com/apache/flink/pull/19225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19253: [hive] support nvl(null, null)
flinkbot commented on pull request #19253: URL: https://github.com/apache/flink/pull/19253#issuecomment-1080395612 ## CI report: * dec3d598857c6bf7f43ba5a744647eea7fbe5f5e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #19225: [FLINK-26799][Runtime/StateBackends] fix StateChangeFormat#read and e…
rkhachatryan commented on pull request #19225: URL: https://github.com/apache/flink/pull/19225#issuecomment-1080396139 Hi @zoltar9264, thank you, I've merged the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-26799) StateChangeFormat#read not seek to offset correctly
[
https://issues.apache.org/jira/browse/FLINK-26799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan resolved FLINK-26799.
---
Resolution: Fixed
Merged into master as d5e472af4f817d343fae9073aad162ee13f08d6a
into 1.15 as 3fed74d757b.
> StateChangeFormat#read not seek to offset correctly
> ---
>
> Key: FLINK-26799
> URL: https://issues.apache.org/jira/browse/FLINK-26799
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Feifan Wang
>Assignee: Feifan Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Attachments: image-2022-03-24-18-12-09-742.png
>
>
> StateChangeFormat#read must seek to offset before read, current implement as
> follows :
>
> {code:java}
> FSDataInputStream stream = handle.openInputStream();
> DataInputViewStreamWrapper input = wrap(stream);
> if (stream.getPos() != offset) {
> LOG.debug("seek from {} to {}", stream.getPos(), offset);
> input.skipBytesToRead((int) offset);
> }{code}
> But the if condition is incorrect, stream.getPos() return the position of
> underlying stream which is different from position of input.
> By the way, because of wrapped by BufferedInputStream, position of underlying
> stream always at n*bufferSize or the end of file.
> Actually, input is aways at position 0 at beginning, so I think we can seek
> to the offset directly.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19253: [hive] support nvl(null, null)
flinkbot edited a comment on pull request #19253: URL: https://github.com/apache/flink/pull/19253#issuecomment-1080395612 ## CI report: * dec3d598857c6bf7f43ba5a744647eea7fbe5f5e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33814) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19225: [FLINK-26799][Runtime/StateBackends] fix StateChangeFormat#read and e…
zoltar9264 commented on pull request #19225: URL: https://github.com/apache/flink/pull/19225#issuecomment-1080402796 My pleasure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26789) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
[
https://issues.apache.org/jira/browse/FLINK-26789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513263#comment-17513263
]
Roman Khachatryan commented on FLINK-26789:
---
I guess the correct link is https://issues.apache.org/jira/browse/FLINK-26882
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
> --
>
> Key: FLINK-26789
> URL: https://issues.apache.org/jira/browse/FLINK-26789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> [This
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=894&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1&l=5687]
> failed due to
> {{RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState}}:
> {code}
> Mar 21 17:05:32 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState
> Time elapsed: 23.966 s <<< FAILURE!
> Mar 21 17:05:32 java.lang.AssertionError: expected:<[(0,24000), (1,22500),
> (0,34500), (1,33000), (0,21000), (0,45000), (2,31500), (2,42000), (1,6000),
> (0,28500), (0,52500), (2,15000), (1,3000), (1,51000), (0,1500), (0,49500),
> (2,12000), (2,6), (0,36000), (1,10500), (1,58500), (0,46500), (0,9000),
> (0,57000), (2,19500), (2,43500), (1,7500), (1,55500), (2,3), (1,18000),
> (0,54000), (2,40500), (1,4500), (0,16500), (2,27000), (1,39000), (2,13500),
> (1,25500), (0,37500), (0,61500), (2,0), (2,48000)]> but was:<[(1,22500),
> (1,33000), (0,21000), (2,18000), (1,6000), (0,20500), (0,52500), (0,15000),
> (0,31000), (2,12000), (2,6), (0,36000), (1,58500), (1,10500), (0,46500),
> (0,25000), (0,41000), (0,9000), (0,57000), (2,43500), (0,3), (1,4500),
> (2,27000), (1,15000), (0,35000), (0,19000), (0,3000), (1,25500), (0,61500),
> (2,48000), (2,0), (0,24000), (0,34500), (0,45000), (2,31500), (1,19500),
> (2,1), (2,42000), (0,12500), (0,28500), (2,15000), (1,3000), (1,51000),
> (0,23000), (0,49500), (0,1500), (0,33000), (0,1000), (2,19500), (1,7500),
> (1,55500), (2,3), (1,18000), (0,6000), (0,38000), (0,54000), (2,40500),
> (0,500), (0,16500), (1,39000), (1,7000), (0,11000), (2,13500), (0,37500)]>
> Mar 21 17:05:32 at org.junit.Assert.fail(Assert.java:89)
> Mar 21 17:05:32 at org.junit.Assert.failNotEquals(Assert.java:835)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:120)
> Mar 21 17:05:32 at org.junit.Assert.assertEquals(Assert.java:146)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:218)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:122)
> Mar 21 17:05:32 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState(RescaleCheckpointManuallyITCase.java:88)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26868) In JobLeaderIdService.java#L36 the annotation may be " Start the service with the given job leader id actions" instead of "Start the service with the given job le
[ https://issues.apache.org/jira/browse/FLINK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513265#comment-17513265 ] Zhuang Liu commented on FLINK-26868: - [Martijn Visser][https://issues.apache.org/jira/secure/ViewProfile.jspa?name=martijnvisser] Thank you for your reply, I will read this official document carefully. > In JobLeaderIdService.java#L36 the annotation may be " Start the service > with the given job leader id actions" instead of "Start the service with the > given job leader actions." > - > > Key: FLINK-26868 > URL: https://issues.apache.org/jira/browse/FLINK-26868 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.14.4 >Reporter: Zhuang Liu >Priority: Minor > > In > https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/JobLeaderIdService.java#L36 >the annotation may be " Start the service with the given job leader id > actions" instead of "Start the service with the given job leader actions." -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (FLINK-26868) In JobLeaderIdService.java#L36 the annotation may be " Start the service with the given job leader id actions" instead of "Start the service with the given job leader actions
[ https://issues.apache.org/jira/browse/FLINK-26868 ] Zhuang Liu deleted comment on FLINK-26868: - was (Author: liu zhuang): [Martijn Visser][https://issues.apache.org/jira/secure/ViewProfile.jspa?name=martijnvisser] Thank you for your reply, I will read this official document carefully. > In JobLeaderIdService.java#L36 the annotation may be " Start the service > with the given job leader id actions" instead of "Start the service with the > given job leader actions." > - > > Key: FLINK-26868 > URL: https://issues.apache.org/jira/browse/FLINK-26868 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.14.4 >Reporter: Zhuang Liu >Priority: Minor > > In > https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/JobLeaderIdService.java#L36 >the annotation may be " Start the service with the given job leader id > actions" instead of "Start the service with the given job leader actions." -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration
[
https://issues.apache.org/jira/browse/FLINK-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-26881:
---
Labels: pull-request-available (was: )
> Stop with savepoint should pick up the targetDirectory from Flink
> configuration
> ---
>
> Key: FLINK-26881
> URL: https://issues.apache.org/jira/browse/FLINK-26881
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: kubernetes-operator-0.1.0
>
>
> Upgrading stateless FlinkDeployment to savepoint still could not work.
> Because {{FlinkService#cancelJob}} does not pick up the target directory in
> the flink configuration.
>
> {code:java}
> case SAVEPOINT:
> String savepoint =
> clusterClient
> .stopWithSavepoint(jobID, false, null)
> .get(1, TimeUnit.MINUTES); {code}
> We should use configured savepoint directory instead of {{null}} in the above
> implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #19108: [FLINK-26675][runtime] Parallelized heavy serialization in StreamingJ…
flinkbot edited a comment on pull request #19108: URL: https://github.com/apache/flink/pull/19108#issuecomment-1068849604 ## CI report: * 7b9c57dda493e9be3c0620cc188d9daf3feba1ff UNKNOWN * 8e3393b1fd8f279c682c3bae4b882e4ddb58da71 UNKNOWN * 8f891ae3b7e0c394d9bfc65fa60ebbc927ca12a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33799) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19200: [FLINK-26281][connectors/elasticsearch] Improve documentation
flinkbot edited a comment on pull request #19200: URL: https://github.com/apache/flink/pull/19200#issuecomment-1075185284 ## CI report: * 5b70ee445d5550303b6c016302b2d4ce9a70d1e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33602) * 976f97ce2aab47c80a6c4db4244561c1483b0be8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
