[GitHub] [flink] flinkbot edited a comment on pull request #17645: [FLINK-24667][runtime] Fix error handling in ChannelStateCheckpointWriter
flinkbot edited a comment on pull request #17645: URL: https://github.com/apache/flink/pull/17645#issuecomment-957822962 ## CI report: * b3688cf3f05ac4ef7ae74acb0c992d3f1d60a340 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25969) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24793) DefaultSchedulerLocalRecoveryITCase fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440264#comment-17440264
]
Till Rohrmann commented on FLINK-24793:
---
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26093=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=ea7cf968-e585-52cb-e0fc-f48de023a7ca=4842
> DefaultSchedulerLocalRecoveryITCase fails on AZP
>
>
> Key: FLINK-24793
> URL: https://issues.apache.org/jira/browse/FLINK-24793
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull}} and
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryRegion}} fails on AZP
> with:
> {code}
> Nov 04 23:01:32 java.lang.IllegalArgumentException: attempt does not exist
> Nov 04 23:01:32 at
> org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex.getPriorExecutionAttempt(ArchivedExecutionVertex.java:109)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.assertNonLocalRecoveredTasksEquals(DefaultSchedulerLocalRecoveryITCase.java:92)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryInternal(DefaultSchedulerLocalRecoveryITCase.java:80)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull(DefaultSchedulerLocalRecoveryITCase.java:65)
> Nov 04 23:01:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 04 23:01:32 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 04 23:01:32 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 04 23:01:32 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 04 23:01:32 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 04 23:01:32 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 04 23:01:32 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Nov 04 23:01:32 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Nov 04 23:01:32 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Nov 04 23:01:32 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Nov 04 23:01:32 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Nov 04 23:01:32 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Nov 04 23:01:32 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Nov 04 23:01:32 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Nov 04 23:01:32 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Nov 04 23:01:32 at
>
[jira] [Commented] (FLINK-24738) Fail during announcing buffer size to released local channel
[
https://issues.apache.org/jira/browse/FLINK-24738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440261#comment-17440261
]
Dawid Wysakowicz commented on FLINK-24738:
--
Merged in:
* master
** 138e83e2bc6feb59ecd839b03491cd8f23ee1c66
> Fail during announcing buffer size to released local channel
>
>
> Key: FLINK-24738
> URL: https://issues.apache.org/jira/browse/FLINK-24738
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.14.0
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Since we can release all resources not only when the mailbox would be
> finished but also from the mailbox:
> {noformat}
> org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.releaseAllResources(LocalInputChannel.java:331)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.transformEvent(SingleInputGate.java:808)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.transformToBufferOrEvent(SingleInputGate.java:757)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:687)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:666)
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:142)
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:150)
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:503)
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:816)
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:768)
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:936)
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:750)
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
> {noformat}
> It is possible that after it the BufferDebloater announce the new buffer size
> which will fail because the channel is released already:
> {noformat}
> Caused by: java.lang.IllegalStateException: Channel released.
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.announceBufferSize(LocalInputChannel.java:354)
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.announceBufferSize(SingleInputGate.java:389)
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.announceBufferSize(InputGateWithMetrics.java:102)
> at
> org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater.recalculateBufferSize(BufferDebloater.java:101)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:801)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$4(StreamTask.java:791)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:816)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:768)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:936)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:750)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> So I think that we should replace `checkState` with `if` for
>
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440260#comment-17440260
]
Till Rohrmann commented on FLINK-24763:
---
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26093=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=13598
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-03T22:10:11.5127349Z Nov 03 22:10:11 at
>
[jira] [Updated] (FLINK-24738) Fail during announcing buffer size to released local channel
[
https://issues.apache.org/jira/browse/FLINK-24738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz updated FLINK-24738:
-
Fix Version/s: 1.15.0
> Fail during announcing buffer size to released local channel
>
>
> Key: FLINK-24738
> URL: https://issues.apache.org/jira/browse/FLINK-24738
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.14.0
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Since we can release all resources not only when the mailbox would be
> finished but also from the mailbox:
> {noformat}
> org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.releaseAllResources(LocalInputChannel.java:331)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.transformEvent(SingleInputGate.java:808)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.transformToBufferOrEvent(SingleInputGate.java:757)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:687)
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:666)
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:142)
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:150)
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:503)
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:816)
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:768)
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:936)
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:750)
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
> {noformat}
> It is possible that after it the BufferDebloater announce the new buffer size
> which will fail because the channel is released already:
> {noformat}
> Caused by: java.lang.IllegalStateException: Channel released.
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> at
> org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.announceBufferSize(LocalInputChannel.java:354)
> at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.announceBufferSize(SingleInputGate.java:389)
> at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.announceBufferSize(InputGateWithMetrics.java:102)
> at
> org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater.recalculateBufferSize(BufferDebloater.java:101)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:801)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$4(StreamTask.java:791)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:816)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:768)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:936)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:750)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> So I think that we should replace `checkState` with `if` for
> LocalInputChannel#announceBufferSize since released channel is expected here.
--
This message
[jira] [Commented] (FLINK-24764) pyflink/table/tests/test_udf.py hang on Azure
[
https://issues.apache.org/jira/browse/FLINK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440257#comment-17440257
]
Till Rohrmann commented on FLINK-24764:
---
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26093=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=22613
> pyflink/table/tests/test_udf.py hang on Azure
> -
>
> Key: FLINK-24764
> URL: https://issues.apache.org/jira/browse/FLINK-24764
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.5
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {{pyflink/table/tests/test_udf.py}} seems to hang on Azure.
> {code}
> 2021-11-04T03:12:01.4537829Z py36-cython run-test: commands[3] | pytest
> --durations=20
> 2021-11-04T03:12:03.6955063Z = test session
> starts ==
> 2021-11-04T03:12:03.6957461Z platform linux -- Python 3.6.13, pytest-6.2.5,
> py-1.10.0, pluggy-1.0.0
> 2021-11-04T03:12:03.6959285Z cachedir: .tox/py36-cython/.pytest_cache
> 2021-11-04T03:12:03.6960653Z rootdir: /__w/1/s/flink-python
> 2021-11-04T03:12:03.6961356Z collected 690 items
> 2021-11-04T03:12:03.6961755Z
> 2021-11-04T03:12:04.6615796Z pyflink/common/tests/test_configuration.py
> ..[ 1%]
> 2021-11-04T03:12:04.9315499Z pyflink/common/tests/test_execution_config.py
> ...[ 4%]
> 2021-11-04T03:12:05.4226061Z
> pyflink/common/tests/test_serialization_schemas.py ... [
> 5%]
> 2021-11-04T03:12:05.8920762Z pyflink/common/tests/test_typeinfo.py ...
> [ 5%]
> 2021-11-04T03:12:10.3843622Z
> pyflink/dataset/tests/test_execution_environment.py ...s.[
> 6%]
> 2021-11-04T03:12:10.4385641Z
> pyflink/dataset/tests/test_execution_environment_completeness.py . [
> 7%]
> 2021-11-04T03:12:10.5390180Z
> pyflink/datastream/tests/test_check_point_config.py ... [
> 8%]
> 2021-11-04T03:12:20.1148835Z pyflink/datastream/tests/test_connectors.py ...
> [ 9%]
> 2021-11-04T03:13:12.4436977Z pyflink/datastream/tests/test_data_stream.py
> ... [ 13%]
> 2021-11-04T03:13:22.6815256Z
> [ 14%]
> 2021-11-04T03:13:22.9777981Z pyflink/datastream/tests/test_state_backend.py
> ..[ 16%]
> 2021-11-04T03:13:33.4281095Z
> pyflink/datastream/tests/test_stream_execution_environment.py .. [
> 18%]
> 2021-11-04T03:13:45.3707210Z .s.
> [ 21%]
> 2021-11-04T03:13:45.5100419Z
> pyflink/datastream/tests/test_stream_execution_environment_completeness.py .
> [ 21%]
> 2021-11-04T03:13:45.5107357Z
> [ 21%]
> 2021-11-04T03:13:45.5824541Z pyflink/fn_execution/tests/test_coders.py
> s [ 24%]
> 2021-11-04T03:13:45.6311670Z pyflink/fn_execution/tests/test_fast_coders.py
> ... [ 27%]
> 2021-11-04T03:13:45.6480686Z
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py . [
> 27%]
> 2021-11-04T03:13:48.3033527Z
> pyflink/fn_execution/tests/test_process_mode_boot.py ... [
> 28%]
> 2021-11-04T03:13:48.3169538Z pyflink/metrics/tests/test_metric.py .
> [ 28%]
> 2021-11-04T03:13:48.3928810Z pyflink/ml/tests/test_ml_environment.py ...
> [ 29%]
> 2021-11-04T03:13:48.4381082Z pyflink/ml/tests/test_ml_environment_factory.py
> ... [ 29%]
> 2021-11-04T03:13:48.4696143Z pyflink/ml/tests/test_params.py .
> [ 31%]
> 2021-11-04T03:13:48.5140301Z pyflink/ml/tests/test_pipeline.py
> [ 32%]
> 2021-11-04T03:13:50.2573824Z pyflink/ml/tests/test_pipeline_it_case.py ...
> [ 32%]
> 2021-11-04T03:13:50.3598135Z pyflink/ml/tests/test_pipeline_stage.py ..
> [ 32%]
> 2021-11-04T03:14:18.5397420Z pyflink/table/tests/test_aggregate.py .
> [ 34%]
> 2021-11-04T03:14:20.1852937Z pyflink/table/tests/test_calc.py ...
> [ 35%]
> 2021-11-04T03:14:21.3674525Z pyflink/table/tests/test_catalog.py
> [ 40%]
> 2021-11-04T03:14:22.4375814Z ...
> [ 46%]
> 2021-11-04T03:14:22.4966492Z
[jira] [Commented] (FLINK-24764) pyflink/table/tests/test_udf.py hang on Azure
[
https://issues.apache.org/jira/browse/FLINK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440258#comment-17440258
]
Till Rohrmann commented on FLINK-24764:
---
cc [~dianfu] can you take a look?
> pyflink/table/tests/test_udf.py hang on Azure
> -
>
> Key: FLINK-24764
> URL: https://issues.apache.org/jira/browse/FLINK-24764
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {{pyflink/table/tests/test_udf.py}} seems to hang on Azure.
> {code}
> 2021-11-04T03:12:01.4537829Z py36-cython run-test: commands[3] | pytest
> --durations=20
> 2021-11-04T03:12:03.6955063Z = test session
> starts ==
> 2021-11-04T03:12:03.6957461Z platform linux -- Python 3.6.13, pytest-6.2.5,
> py-1.10.0, pluggy-1.0.0
> 2021-11-04T03:12:03.6959285Z cachedir: .tox/py36-cython/.pytest_cache
> 2021-11-04T03:12:03.6960653Z rootdir: /__w/1/s/flink-python
> 2021-11-04T03:12:03.6961356Z collected 690 items
> 2021-11-04T03:12:03.6961755Z
> 2021-11-04T03:12:04.6615796Z pyflink/common/tests/test_configuration.py
> ..[ 1%]
> 2021-11-04T03:12:04.9315499Z pyflink/common/tests/test_execution_config.py
> ...[ 4%]
> 2021-11-04T03:12:05.4226061Z
> pyflink/common/tests/test_serialization_schemas.py ... [
> 5%]
> 2021-11-04T03:12:05.8920762Z pyflink/common/tests/test_typeinfo.py ...
> [ 5%]
> 2021-11-04T03:12:10.3843622Z
> pyflink/dataset/tests/test_execution_environment.py ...s.[
> 6%]
> 2021-11-04T03:12:10.4385641Z
> pyflink/dataset/tests/test_execution_environment_completeness.py . [
> 7%]
> 2021-11-04T03:12:10.5390180Z
> pyflink/datastream/tests/test_check_point_config.py ... [
> 8%]
> 2021-11-04T03:12:20.1148835Z pyflink/datastream/tests/test_connectors.py ...
> [ 9%]
> 2021-11-04T03:13:12.4436977Z pyflink/datastream/tests/test_data_stream.py
> ... [ 13%]
> 2021-11-04T03:13:22.6815256Z
> [ 14%]
> 2021-11-04T03:13:22.9777981Z pyflink/datastream/tests/test_state_backend.py
> ..[ 16%]
> 2021-11-04T03:13:33.4281095Z
> pyflink/datastream/tests/test_stream_execution_environment.py .. [
> 18%]
> 2021-11-04T03:13:45.3707210Z .s.
> [ 21%]
> 2021-11-04T03:13:45.5100419Z
> pyflink/datastream/tests/test_stream_execution_environment_completeness.py .
> [ 21%]
> 2021-11-04T03:13:45.5107357Z
> [ 21%]
> 2021-11-04T03:13:45.5824541Z pyflink/fn_execution/tests/test_coders.py
> s [ 24%]
> 2021-11-04T03:13:45.6311670Z pyflink/fn_execution/tests/test_fast_coders.py
> ... [ 27%]
> 2021-11-04T03:13:45.6480686Z
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py . [
> 27%]
> 2021-11-04T03:13:48.3033527Z
> pyflink/fn_execution/tests/test_process_mode_boot.py ... [
> 28%]
> 2021-11-04T03:13:48.3169538Z pyflink/metrics/tests/test_metric.py .
> [ 28%]
> 2021-11-04T03:13:48.3928810Z pyflink/ml/tests/test_ml_environment.py ...
> [ 29%]
> 2021-11-04T03:13:48.4381082Z pyflink/ml/tests/test_ml_environment_factory.py
> ... [ 29%]
> 2021-11-04T03:13:48.4696143Z pyflink/ml/tests/test_params.py .
> [ 31%]
> 2021-11-04T03:13:48.5140301Z pyflink/ml/tests/test_pipeline.py
> [ 32%]
> 2021-11-04T03:13:50.2573824Z pyflink/ml/tests/test_pipeline_it_case.py ...
> [ 32%]
> 2021-11-04T03:13:50.3598135Z pyflink/ml/tests/test_pipeline_stage.py ..
> [ 32%]
> 2021-11-04T03:14:18.5397420Z pyflink/table/tests/test_aggregate.py .
> [ 34%]
> 2021-11-04T03:14:20.1852937Z pyflink/table/tests/test_calc.py ...
> [ 35%]
> 2021-11-04T03:14:21.3674525Z pyflink/table/tests/test_catalog.py
> [ 40%]
> 2021-11-04T03:14:22.4375814Z ...
> [ 46%]
> 2021-11-04T03:14:22.4966492Z pyflink/table/tests/test_catalog_completeness.py
> [ 47%]
> 2021-11-04T03:14:22.7401730Z
[jira] [Updated] (FLINK-24764) pyflink/table/tests/test_udf.py hang on Azure
[
https://issues.apache.org/jira/browse/FLINK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24764:
--
Affects Version/s: 1.15.0
> pyflink/table/tests/test_udf.py hang on Azure
> -
>
> Key: FLINK-24764
> URL: https://issues.apache.org/jira/browse/FLINK-24764
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {{pyflink/table/tests/test_udf.py}} seems to hang on Azure.
> {code}
> 2021-11-04T03:12:01.4537829Z py36-cython run-test: commands[3] | pytest
> --durations=20
> 2021-11-04T03:12:03.6955063Z = test session
> starts ==
> 2021-11-04T03:12:03.6957461Z platform linux -- Python 3.6.13, pytest-6.2.5,
> py-1.10.0, pluggy-1.0.0
> 2021-11-04T03:12:03.6959285Z cachedir: .tox/py36-cython/.pytest_cache
> 2021-11-04T03:12:03.6960653Z rootdir: /__w/1/s/flink-python
> 2021-11-04T03:12:03.6961356Z collected 690 items
> 2021-11-04T03:12:03.6961755Z
> 2021-11-04T03:12:04.6615796Z pyflink/common/tests/test_configuration.py
> ..[ 1%]
> 2021-11-04T03:12:04.9315499Z pyflink/common/tests/test_execution_config.py
> ...[ 4%]
> 2021-11-04T03:12:05.4226061Z
> pyflink/common/tests/test_serialization_schemas.py ... [
> 5%]
> 2021-11-04T03:12:05.8920762Z pyflink/common/tests/test_typeinfo.py ...
> [ 5%]
> 2021-11-04T03:12:10.3843622Z
> pyflink/dataset/tests/test_execution_environment.py ...s.[
> 6%]
> 2021-11-04T03:12:10.4385641Z
> pyflink/dataset/tests/test_execution_environment_completeness.py . [
> 7%]
> 2021-11-04T03:12:10.5390180Z
> pyflink/datastream/tests/test_check_point_config.py ... [
> 8%]
> 2021-11-04T03:12:20.1148835Z pyflink/datastream/tests/test_connectors.py ...
> [ 9%]
> 2021-11-04T03:13:12.4436977Z pyflink/datastream/tests/test_data_stream.py
> ... [ 13%]
> 2021-11-04T03:13:22.6815256Z
> [ 14%]
> 2021-11-04T03:13:22.9777981Z pyflink/datastream/tests/test_state_backend.py
> ..[ 16%]
> 2021-11-04T03:13:33.4281095Z
> pyflink/datastream/tests/test_stream_execution_environment.py .. [
> 18%]
> 2021-11-04T03:13:45.3707210Z .s.
> [ 21%]
> 2021-11-04T03:13:45.5100419Z
> pyflink/datastream/tests/test_stream_execution_environment_completeness.py .
> [ 21%]
> 2021-11-04T03:13:45.5107357Z
> [ 21%]
> 2021-11-04T03:13:45.5824541Z pyflink/fn_execution/tests/test_coders.py
> s [ 24%]
> 2021-11-04T03:13:45.6311670Z pyflink/fn_execution/tests/test_fast_coders.py
> ... [ 27%]
> 2021-11-04T03:13:45.6480686Z
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py . [
> 27%]
> 2021-11-04T03:13:48.3033527Z
> pyflink/fn_execution/tests/test_process_mode_boot.py ... [
> 28%]
> 2021-11-04T03:13:48.3169538Z pyflink/metrics/tests/test_metric.py .
> [ 28%]
> 2021-11-04T03:13:48.3928810Z pyflink/ml/tests/test_ml_environment.py ...
> [ 29%]
> 2021-11-04T03:13:48.4381082Z pyflink/ml/tests/test_ml_environment_factory.py
> ... [ 29%]
> 2021-11-04T03:13:48.4696143Z pyflink/ml/tests/test_params.py .
> [ 31%]
> 2021-11-04T03:13:48.5140301Z pyflink/ml/tests/test_pipeline.py
> [ 32%]
> 2021-11-04T03:13:50.2573824Z pyflink/ml/tests/test_pipeline_it_case.py ...
> [ 32%]
> 2021-11-04T03:13:50.3598135Z pyflink/ml/tests/test_pipeline_stage.py ..
> [ 32%]
> 2021-11-04T03:14:18.5397420Z pyflink/table/tests/test_aggregate.py .
> [ 34%]
> 2021-11-04T03:14:20.1852937Z pyflink/table/tests/test_calc.py ...
> [ 35%]
> 2021-11-04T03:14:21.3674525Z pyflink/table/tests/test_catalog.py
> [ 40%]
> 2021-11-04T03:14:22.4375814Z ...
> [ 46%]
> 2021-11-04T03:14:22.4966492Z pyflink/table/tests/test_catalog_completeness.py
> [ 47%]
> 2021-11-04T03:14:22.7401730Z pyflink/table/tests/test_column_operation.py
> [
[GitHub] [flink] flinkbot edited a comment on pull request #17700: [FLINK-24773][kafka] Fail job if unhandled exception occurs during committing
flinkbot edited a comment on pull request #17700: URL: https://github.com/apache/flink/pull/17700#issuecomment-961972204 ## CI report: * 228e5a9f2d227c5311ad12af866a16c8ba58c80b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26048) * 2d52087c3d86c341ccd0911e06d336a31d22daa0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26115) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #17663: [FLINK-24738][runtime] Ignoring buffer size announcement if the channel is released already
dawidwys merged pull request #17663: URL: https://github.com/apache/flink/pull/17663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17645: [FLINK-24667][runtime] Fix error handling in ChannelStateCheckpointWriter
flinkbot edited a comment on pull request #17645: URL: https://github.com/apache/flink/pull/17645#issuecomment-957822962 ## CI report: * b3688cf3f05ac4ef7ae74acb0c992d3f1d60a340 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25969) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17653: FLINK SQL checkpoint不生效
flinkbot edited a comment on pull request #17653: URL: https://github.com/apache/flink/pull/17653#issuecomment-958686553 ## CI report: * 237fb3a04e81602238b87ffc24423fc6f4cfe0ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26063) * 3b5b5428271b0e0fad1343faeffb5416bbdec3a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26114) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24806:
--
Fix Version/s: 1.14.1
1.13.4
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.14.0, 1.13.3, 1.15.0
>Reporter: Yun Gao
>Assignee: Michał Ciesielczyk
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
[jira] [Updated] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24806:
--
Affects Version/s: 1.14.0
1.13.3
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.14.0, 1.13.3, 1.15.0
>Reporter: Yun Gao
>Assignee: Michał Ciesielczyk
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
[jira] [Assigned] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-24806:
-
Assignee: Michał Ciesielczyk
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Assignee: Michał Ciesielczyk
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
[jira] [Updated] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24806:
--
Fix Version/s: 1.15.0
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Assignee: Michał Ciesielczyk
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
[jira] [Comment Edited] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440255#comment-17440255
]
Till Rohrmann edited comment on FLINK-24806 at 11/8/21, 7:54 AM:
-
[~cmick] sure. A fix will be highly appreciated by the community. I'll assign
the ticket to you.
was (Author: till.rohrmann):
[~cmick] sure, I'll assign the ticket to you.
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
>
[jira] [Commented] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440255#comment-17440255
]
Till Rohrmann commented on FLINK-24806:
---
[~cmick] sure, I'll assign the ticket to you.
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
[GitHub] [flink] dawidwys commented on pull request #17645: [FLINK-24667][runtime] Fix error handling in ChannelStateCheckpointWriter
dawidwys commented on pull request #17645: URL: https://github.com/apache/flink/pull/17645#issuecomment-962894828 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26106) * 2b217db677c48effdbc3609657c7bbbdc46bf81a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26110) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17700: [FLINK-24773][kafka] Fail job if unhandled exception occurs during committing
flinkbot edited a comment on pull request #17700: URL: https://github.com/apache/flink/pull/17700#issuecomment-961972204 ## CI report: * 228e5a9f2d227c5311ad12af866a16c8ba58c80b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26048) * 2d52087c3d86c341ccd0911e06d336a31d22daa0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24806) RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due to connection error
[
https://issues.apache.org/jira/browse/FLINK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440254#comment-17440254
]
Till Rohrmann commented on FLINK-24806:
---
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26093=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=13479
> RMQSourceITCase.testStopWithSavepoint and testAckFailure failed on azure due
> to connection error
>
>
> Key: FLINK-24806
> URL: https://issues.apache.org/jira/browse/FLINK-24806
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Nov 05 22:12:57 [INFO] Running
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0,
> Time elapsed: 20.921 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase
> Nov 05 22:13:18 [ERROR] testStopWithSavepoint Time elapsed: 3.39 s <<<
> ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 05 22:13:18 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 05 22:13:18 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 05 22:13:18 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 05 22:13:18
> Nov 05 22:13:18 [ERROR] testAckFailure Time elapsed: 1.842 s <<< ERROR!
> Nov 05 22:13:18 java.io.IOException
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:396)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
> Nov 05 22:13:18 at
> com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1130)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1087)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1045)
> Nov 05 22:13:18 at
> com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1207)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.getRMQConnection(RMQSourceITCase.java:203)
> Nov 05 22:13:18 at
> org.apache.flink.streaming.connectors.rabbitmq.RMQSourceITCase.setUp(RMQSourceITCase.java:98)
> Nov 05 22:13:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 05 22:13:18 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 05 22:13:18 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17653: FLINK SQL checkpoint不生效
flinkbot edited a comment on pull request #17653: URL: https://github.com/apache/flink/pull/17653#issuecomment-958686553 ## CI report: * 237fb3a04e81602238b87ffc24423fc6f4cfe0ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26063) * 3b5b5428271b0e0fad1343faeffb5416bbdec3a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24401) TM cannot exit after Metaspace OOM
[
https://issues.apache.org/jira/browse/FLINK-24401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440253#comment-17440253
]
Piotr Nowojski edited comment on FLINK-24401 at 11/8/21, 7:52 AM:
--
Merged to master/release-1.14/release-1.13 as
365d12d499a/8f73e6c38cc/3b5b5428271 respectively
Thanks again [~fanrui] for reporting and fixing the issue!
was (Author: pnowojski):
Merged to master/release-1.14/release-1.13 as
365d12d499a/8f73e6c38cc/3b5b5428271 respectively
> TM cannot exit after Metaspace OOM
> --
>
> Key: FLINK-24401
> URL: https://issues.apache.org/jira/browse/FLINK-24401
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / Task
>Affects Versions: 1.12.0, 1.13.0
>Reporter: future
>Assignee: future
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
> Attachments: image-2021-09-29-12-00-28-510.png,
> image-2021-09-29-12-00-44-812.png
>
>
> Hi masters, from the code and log, we can see that OOM will terminateJVM
> directly, but Metaspace OutOfMemoryError will graceful shutdown. The code
> comment mentions: {{_it does not usually require more class loading to fail
> again with the Metaspace OutOfMemoryError_.}}.
> But we encountered: after Metaspace OutOfMemoryError,
> {{_java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner$Result_.}}, makes Tm
> unable to exit, keeps trying again, keeps NoClassDefFoundError, keeps class
> loading failure, until kill tm by manually.
> I want to add a catch Throwable in the onFatalError method, and directly
> terminateJVM() in the catch. Is there any problem with this strategy?
>
> [code link
> |https://github.com/apache/flink/blob/4fe9f525a92319acc1e3434bebed601306f7a16f/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java#L312]
> picture:
>
> !image-2021-09-29-12-00-44-812.png|width=1337,height=692!
> !image-2021-09-29-12-00-28-510.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-24401) TM cannot exit after Metaspace OOM
[
https://issues.apache.org/jira/browse/FLINK-24401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-24401.
--
Fix Version/s: 1.15.0
Resolution: Fixed
Merged to master/release-1.14/release-1.13 as
365d12d499a/8f73e6c38cc/3b5b5428271 respectively
> TM cannot exit after Metaspace OOM
> --
>
> Key: FLINK-24401
> URL: https://issues.apache.org/jira/browse/FLINK-24401
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / Task
>Affects Versions: 1.12.0, 1.13.0
>Reporter: future
>Assignee: future
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
> Attachments: image-2021-09-29-12-00-28-510.png,
> image-2021-09-29-12-00-44-812.png
>
>
> Hi masters, from the code and log, we can see that OOM will terminateJVM
> directly, but Metaspace OutOfMemoryError will graceful shutdown. The code
> comment mentions: {{_it does not usually require more class loading to fail
> again with the Metaspace OutOfMemoryError_.}}.
> But we encountered: after Metaspace OutOfMemoryError,
> {{_java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner$Result_.}}, makes Tm
> unable to exit, keeps trying again, keeps NoClassDefFoundError, keeps class
> loading failure, until kill tm by manually.
> I want to add a catch Throwable in the onFatalError method, and directly
> terminateJVM() in the catch. Is there any problem with this strategy?
>
> [code link
> |https://github.com/apache/flink/blob/4fe9f525a92319acc1e3434bebed601306f7a16f/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java#L312]
> picture:
>
> !image-2021-09-29-12-00-44-812.png|width=1337,height=692!
> !image-2021-09-29-12-00-28-510.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] pnowojski merged pull request #17668: [FLINK-24401][runtime] Fix the bug of TM cannot exit after Metaspace OOM
pnowojski merged pull request #17668: URL: https://github.com/apache/flink/pull/17668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24077) HBaseConnectorITCase.testTableSink
[
https://issues.apache.org/jira/browse/FLINK-24077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440243#comment-17440243
]
Till Rohrmann commented on FLINK-24077:
---
cc [~arvid], [~twalthr]
> HBaseConnectorITCase.testTableSink
> --
>
> Key: FLINK-24077
> URL: https://issues.apache.org/jira/browse/FLINK-24077
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23160=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=12962
> {code}
> Aug 31 05:10:58 Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 73.758 sec <<< FAILURE! - in
> org.apache.flink.connector.hbase2.HBaseConnectorITCase
> Aug 31 05:10:58
> testTableSink(org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time
> elapsed: 6.516 sec <<< FAILURE!
> Aug 31 05:10:58 java.lang.AssertionError:
> Aug 31 05:10:58 Different elements in arrays: expected 8 elements and
> received 3
> Aug 31 05:10:58 expected: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3], +I[4, 40, null, 400, 4.04, true, Welt-4], +I[5, 50, Hello-5,
> 500, 5.05, false, Welt-5], +I[6, 60, Hello-6, 600, 6.06, true, Welt-6], +I[7,
> 70, Hello-7, 700, 7.07, false, Welt-7], +I[8, 80, null, 800, 8.08, true,
> Welt-8]]
> Aug 31 05:10:58 received: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3]] expected:<8> but was:<3>
> Aug 31 05:10:58 at org.junit.Assert.fail(Assert.java:89)
> Aug 31 05:10:58 at org.junit.Assert.failNotEquals(Assert.java:835)
> Aug 31 05:10:58 at org.junit.Assert.assertEquals(Assert.java:647)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResult(TestBaseUtils.java:395)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResultAsText(TestBaseUtils.java:347)
> Aug 31 05:10:58 at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSink(HBaseConnectorITCase.java:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24789) IllegalStateException with CheckpointCleaner being closed already
[
https://issues.apache.org/jira/browse/FLINK-24789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440241#comment-17440241
]
Till Rohrmann commented on FLINK-24789:
---
What's the progress on this ticket [~dmvk]?
> IllegalStateException with CheckpointCleaner being closed already
> -
>
> Key: FLINK-24789
> URL: https://issues.apache.org/jira/browse/FLINK-24789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Matthias
>Assignee: David Morávek
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
> Attachments:
> logs-ci_build-test_ci_build_finegrained_resource_management-1635785399.zip
>
>
> We experienced a failure of {{OperatorCoordinatorSchedulerTest}} in our VVP
> Fork of Flink. The {{finegrained_resource_management}} test run failed with
> an non-0 exit code:
> {code}
> Nov 01 17:19:12 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on
> project flink-runtime: There are test failures.
> Nov 01 17:19:12 [ERROR]
> Nov 01 17:19:12 [ERROR] Please refer to
> /__w/1/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Nov 01 17:19:12 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Nov 01 17:19:12 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Nov 01 17:19:12 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/1/s/flink-runtime/target/surefire/surefirebooter6007815607334336440.jar
> /__w/1/s/flink-runtime/target/surefire 2021-11-01T16-51-51_363-jvmRun2
> surefire6448660128033443499tmp surefire_4131168043975619749001tmp
> Nov 01 17:19:12 [ERROR] Error occurred in starting fork, check output in log
> Nov 01 17:19:12 [ERROR] Process Exit Code: 239
> Nov 01 17:19:12 [ERROR] Crashed tests:
> Nov 01 17:19:12 [ERROR]
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorSchedulerTest
> Nov 01 17:19:12 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Nov 01 17:19:12 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/1/s/flink-runtime/target/surefire/surefirebooter6007815607334336440.jar
> /__w/1/s/flink-runtime/target/surefire 2021-11-01T16-51-51_363-jvmRun2
> surefire6448660128033443499tmp surefire_4131168043975619749001tmp
> Nov 01 17:19:12 [ERROR] Error occurred in starting fork, check output in log
> Nov 01 17:19:12 [ERROR] Process Exit Code: 239
> Nov 01 17:19:12 [ERROR] Crashed tests:
> Nov 01 17:19:12 [ERROR]
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorSchedulerTest
> Nov 01 17:19:12 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> Nov 01 17:19:12 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> {code}
> It looks like the {{testSnapshotAsyncFailureFailsCheckpoint}} caused it even
> though finishing successfully due to a fatal error when shutting down the
> cluster:
> {code}
> 17:07:27,264 [Checkpoint Timer] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'Checkpoint Timer' produced an uncaught exception. Stopping the
> process...
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> CheckpointsCleaner has already been closed
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:626)
> ~[classes/:?]
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> [?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> [?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:814)
> [?:1.8.0_292]
> at
>
[jira] [Commented] (FLINK-24457) FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover fails with NoSuchElement
[
https://issues.apache.org/jira/browse/FLINK-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440242#comment-17440242
]
Till Rohrmann commented on FLINK-24457:
---
[~xmarker] can you provide a fix for the problem?
> FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover
> fails with NoSuchElement
> -
>
> Key: FLINK-24457
> URL: https://issues.apache.org/jira/browse/FLINK-24457
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.13.3, 1.15.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24781=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=2c68b137-b01d-55c9-e603-3ff3f320364b=23849
> {code}
> Oct 06 00:07:54 [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 41.682 s <<< FAILURE! - in
> org.apache.flink.connector.file.src.FileSourceTextLinesITCase
> Oct 06 00:07:54 [ERROR] testContinuousTextFileSourceWithJobManagerFailover
> Time elapsed: 10.826 s <<< ERROR!
> Oct 06 00:07:54 java.util.NoSuchElementException
> Oct 06 00:07:54 at java.util.LinkedList.removeLast(LinkedList.java:283)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.revert(AbstractCollectResultBuffer.java:112)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.CheckpointedCollectResultBuffer.sinkRestarted(CheckpointedCollectResultBuffer.java:37)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.dealWithResponse(AbstractCollectResultBuffer.java:87)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:147)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Oct 06 00:07:54 at
> org.apache.flink.streaming.api.datastream.DataStreamUtils.collectRecordsFromUnboundedStream(DataStreamUtils.java:142)
> Oct 06 00:07:54 at
> org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSource(FileSourceTextLinesITCase.java:224)
> Oct 06 00:07:54 at
> org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover(FileSourceTextLinesITCase.java:180)
> Oct 06 00:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Oct 06 00:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 06 00:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 06 00:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 06 00:07:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Oct 06 00:07:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 06 00:07:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Oct 06 00:07:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 06 00:07:54 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Oct 06 00:07:54 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Oct 06 00:07:54 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Oct 06 00:07:54 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Oct 06 00:07:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Oct 06 00:07:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Oct 06 00:07:54 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
[jira] [Commented] (FLINK-24763) ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440238#comment-17440238
]
Till Rohrmann commented on FLINK-24763:
---
cc [~arvid]
> ParquetFileSystemITCase.testLimitableBulkFormat failed on Azure
> ---
>
> Key: FLINK-24763
> URL: https://issues.apache.org/jira/browse/FLINK-24763
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> The test {{ParquetFileSystemITCase.testLimitableBulkFormat}} fails with
> {code}
> 2021-11-03T22:10:11.5106075Z Nov 03 22:10:11 [ERROR]
> testLimitableBulkFormat[false] Time elapsed: 9.177 s <<< ERROR!
> 2021-11-03T22:10:11.5106643Z Nov 03 22:10:11 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-11-03T22:10:11.5107213Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-11-03T22:10:11.5111034Z Nov 03 22:10:11 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-11-03T22:10:11.5112190Z Nov 03 22:10:11 at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:188)
> 2021-11-03T22:10:11.5112892Z Nov 03 22:10:11 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2021-11-03T22:10:11.5113393Z Nov 03 22:10:11 at
> org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
> 2021-11-03T22:10:11.5114157Z Nov 03 22:10:11 at
> org.apache.flink.formats.parquet.ParquetFileSystemITCase.testLimitableBulkFormat(ParquetFileSystemITCase.java:128)
> 2021-11-03T22:10:11.5114951Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-11-03T22:10:11.5115568Z Nov 03 22:10:11 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-11-03T22:10:11.5116115Z Nov 03 22:10:11 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-11-03T22:10:11.5116591Z Nov 03 22:10:11 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-11-03T22:10:11.5117088Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-11-03T22:10:11.5117807Z Nov 03 22:10:11 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-11-03T22:10:11.5118821Z Nov 03 22:10:11 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-11-03T22:10:11.5119417Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-11-03T22:10:11.5119944Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-11-03T22:10:11.5120427Z Nov 03 22:10:11 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-11-03T22:10:11.5120919Z Nov 03 22:10:11 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-11-03T22:10:11.5121571Z Nov 03 22:10:11 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-11-03T22:10:11.5122526Z Nov 03 22:10:11 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-11-03T22:10:11.5123245Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-11-03T22:10:11.5123804Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-11-03T22:10:11.5124314Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-11-03T22:10:11.5124806Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-11-03T22:10:11.5125313Z Nov 03 22:10:11 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-11-03T22:10:11.5125810Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-11-03T22:10:11.5126281Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-11-03T22:10:11.5126739Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-11-03T22:10:11.5127349Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-11-03T22:10:11.5128092Z Nov 03 22:10:11 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>
[jira] [Updated] (FLINK-18808) Task-level numRecordsOut metric may be underestimated
[
https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-18808:
---
Labels: pull-request-available usability (was: auto-deprioritized-major
pull-request-available stale-minor usability)
> Task-level numRecordsOut metric may be underestimated
> -
>
> Key: FLINK-18808
> URL: https://issues.apache.org/jira/browse/FLINK-18808
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / Task
>Affects Versions: 1.11.1
>Reporter: ming li
>Priority: Not a Priority
> Labels: pull-request-available, usability
> Attachments: image-2020-08-04-11-28-13-800.png,
> image-2020-08-04-11-32-20-678.png, image-2020-08-13-18-36-13-282.png
>
>
> At present, we only register task-level numRecordsOut metric by reusing
> operator output record counter at the end of OperatorChain.
> {code:java}
> if (config.isChainEnd()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }
> {code}
> If we only send data out through the last operator of OperatorChain, there is
> no problem with this statistics. But consider the following scenario:
> !image-2020-08-04-11-28-13-800.png|width=507,height=174!
> In this JobGraph, we not only send data in the last operator, but also send
> data in the middle operator of OperatorChain (the map operator just returns
> the original value directly). Below is one of our test topology, we can see
> that the statistics actually only have half of the total data received by the
> downstream.
> !image-2020-08-04-11-32-20-678.png|width=648,height=251!
> I think the data sent out by the intermediate operator should also be counted
> into the numRecordsOut of the Task. But currently we are not reusing
> operators output record counters in the intermediate operators, which leads
> to our task-level numRecordsOut metric is underestimated (although this has
> no effect on the actual operation of the job, it may affect our monitoring).
> A simple idea of mine is to modify the condition of reusing operators
> output record counter:
> {code:java}
> if (!config.getNonChainedOutputs(getUserCodeClassloader()).isEmpty()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }{code}
> In addition, I have another question: If a record is broadcast to all
> downstream, should the numRecordsOut counter increase by one or the
> downstream number? It seems that currently we are adding one to calculate the
> numRecordsOut metric.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440240#comment-17440240
]
Till Rohrmann commented on FLINK-24495:
---
[~dianfu] can you take a look?
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-22869) SQLClientSchemaRegistryITCase timeouts on azure
[
https://issues.apache.org/jira/browse/FLINK-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440237#comment-17440237
]
Till Rohrmann commented on FLINK-22869:
---
cc [~twalthr] this test seems to fail quite regularly.
> SQLClientSchemaRegistryITCase timeouts on azure
> ---
>
> Key: FLINK-22869
> URL: https://issues.apache.org/jira/browse/FLINK-22869
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem, Tests
>Affects Versions: 1.12.4, 1.13.2
>Reporter: Xintong Song
>Priority: Critical
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.12.6, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18652=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729=27324
> {code}
> Jun 03 23:51:30 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 227.425 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase
> Jun 03 23:51:30 [ERROR]
> testReading(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 194.931 s <<< ERROR!
> Jun 03 23:51:30 org.junit.runners.model.TestTimedOutException: test timed out
> after 12 milliseconds
> Jun 03 23:51:30 at java.lang.Object.wait(Native Method)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1252)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1326)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.KafkaContainerClient.createTopic(KafkaContainerClient.java:71)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testReading(SQLClientSchemaRegistryITCase.java:102)
> Jun 03 23:51:30 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jun 03 23:51:30 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jun 03 23:51:30 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jun 03 23:51:30 at java.lang.reflect.Method.invoke(Method.java:498)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Jun 03 23:51:30 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Jun 03 23:51:30 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jun 03 23:51:30 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24495:
--
Priority: Critical (was: Major)
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-18808) Task-level numRecordsOut metric may be underestimated
[
https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-18808:
---
Priority: Not a Priority (was: Minor)
> Task-level numRecordsOut metric may be underestimated
> -
>
> Key: FLINK-18808
> URL: https://issues.apache.org/jira/browse/FLINK-18808
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / Task
>Affects Versions: 1.11.1
>Reporter: ming li
>Priority: Not a Priority
> Labels: auto-deprioritized-major, pull-request-available,
> stale-minor, usability
> Attachments: image-2020-08-04-11-28-13-800.png,
> image-2020-08-04-11-32-20-678.png, image-2020-08-13-18-36-13-282.png
>
>
> At present, we only register task-level numRecordsOut metric by reusing
> operator output record counter at the end of OperatorChain.
> {code:java}
> if (config.isChainEnd()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }
> {code}
> If we only send data out through the last operator of OperatorChain, there is
> no problem with this statistics. But consider the following scenario:
> !image-2020-08-04-11-28-13-800.png|width=507,height=174!
> In this JobGraph, we not only send data in the last operator, but also send
> data in the middle operator of OperatorChain (the map operator just returns
> the original value directly). Below is one of our test topology, we can see
> that the statistics actually only have half of the total data received by the
> downstream.
> !image-2020-08-04-11-32-20-678.png|width=648,height=251!
> I think the data sent out by the intermediate operator should also be counted
> into the numRecordsOut of the Task. But currently we are not reusing
> operators output record counters in the intermediate operators, which leads
> to our task-level numRecordsOut metric is underestimated (although this has
> no effect on the actual operation of the job, it may affect our monitoring).
> A simple idea of mine is to modify the condition of reusing operators
> output record counter:
> {code:java}
> if (!config.getNonChainedOutputs(getUserCodeClassloader()).isEmpty()) {
>operatorMetricGroup.getIOMetricGroup().reuseOutputMetricsForTask();
> }{code}
> In addition, I have another question: If a record is broadcast to all
> downstream, should the numRecordsOut counter increase by one or the
> downstream number? It seems that currently we are adding one to calculate the
> numRecordsOut metric.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22869) SQLClientSchemaRegistryITCase timeouts on azure
[
https://issues.apache.org/jira/browse/FLINK-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-22869:
--
Component/s: Table SQL / Ecosystem
> SQLClientSchemaRegistryITCase timeouts on azure
> ---
>
> Key: FLINK-22869
> URL: https://issues.apache.org/jira/browse/FLINK-22869
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem, Tests
>Affects Versions: 1.12.4, 1.13.2
>Reporter: Xintong Song
>Priority: Critical
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.12.6, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18652=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729=27324
> {code}
> Jun 03 23:51:30 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 227.425 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase
> Jun 03 23:51:30 [ERROR]
> testReading(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 194.931 s <<< ERROR!
> Jun 03 23:51:30 org.junit.runners.model.TestTimedOutException: test timed out
> after 12 milliseconds
> Jun 03 23:51:30 at java.lang.Object.wait(Native Method)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1252)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1326)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.KafkaContainerClient.createTopic(KafkaContainerClient.java:71)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testReading(SQLClientSchemaRegistryITCase.java:102)
> Jun 03 23:51:30 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jun 03 23:51:30 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jun 03 23:51:30 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jun 03 23:51:30 at java.lang.reflect.Method.invoke(Method.java:498)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Jun 03 23:51:30 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Jun 03 23:51:30 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jun 03 23:51:30 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-23848) PulsarSourceITCase is failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-23848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440236#comment-17440236
]
Till Rohrmann commented on FLINK-23848:
---
cc [~arvid]
> PulsarSourceITCase is failed on Azure
> -
>
> Key: FLINK-23848
> URL: https://issues.apache.org/jira/browse/FLINK-23848
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Jark Wu
>Assignee: Yufan Sheng
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22412=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
> {code}
> 2021-08-17T20:11:53.7228789Z Aug 17 20:11:53 [INFO] Running
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase
> 2021-08-17T20:17:38.2429467Z Aug 17 20:17:38 [ERROR] Tests run: 8, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 344.515 s <<< FAILURE! - in
> org.apache.flink.connector.pulsar.source.PulsarSourceITCase
> 2021-08-17T20:17:38.2430693Z Aug 17 20:17:38 [ERROR]
> testMultipleSplits{TestEnvironment, ExternalContext}[2] Time elapsed: 66.766
> s <<< ERROR!
> 2021-08-17T20:17:38.2431387Z Aug 17 20:17:38 java.lang.RuntimeException:
> Failed to fetch next result
> 2021-08-17T20:17:38.2432035Z Aug 17 20:17:38 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> 2021-08-17T20:17:38.2433345Z Aug 17 20:17:38 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-08-17T20:17:38.2434175Z Aug 17 20:17:38 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:151)
> 2021-08-17T20:17:38.2435028Z Aug 17 20:17:38 at
> org.apache.flink.connectors.test.common.utils.TestDataMatchers$MultipleSplitDataMatcher.matchesSafely(TestDataMatchers.java:133)
> 2021-08-17T20:17:38.2438387Z Aug 17 20:17:38 at
> org.hamcrest.TypeSafeDiagnosingMatcher.matches(TypeSafeDiagnosingMatcher.java:55)
> 2021-08-17T20:17:38.2439100Z Aug 17 20:17:38 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:12)
> 2021-08-17T20:17:38.2439708Z Aug 17 20:17:38 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> 2021-08-17T20:17:38.2440299Z Aug 17 20:17:38 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testMultipleSplits(SourceTestSuiteBase.java:156)
> 2021-08-17T20:17:38.2441007Z Aug 17 20:17:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-08-17T20:17:38.2441526Z Aug 17 20:17:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-08-17T20:17:38.2442068Z Aug 17 20:17:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-08-17T20:17:38.2442759Z Aug 17 20:17:38 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-08-17T20:17:38.2443247Z Aug 17 20:17:38 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
> 2021-08-17T20:17:38.2443812Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2021-08-17T20:17:38.241Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2021-08-17T20:17:38.2445101Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2021-08-17T20:17:38.2445688Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2021-08-17T20:17:38.2446328Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
> 2021-08-17T20:17:38.2447303Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2021-08-17T20:17:38.2448336Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2021-08-17T20:17:38.2448999Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2021-08-17T20:17:38.2449689Z Aug 17 20:17:38 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2021-08-17T20:17:38.2450363Z Aug 17 20:17:38 at
>
[jira] [Updated] (FLINK-22869) SQLClientSchemaRegistryITCase timeouts on azure
[
https://issues.apache.org/jira/browse/FLINK-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-22869:
--
Priority: Critical (was: Major)
> SQLClientSchemaRegistryITCase timeouts on azure
> ---
>
> Key: FLINK-22869
> URL: https://issues.apache.org/jira/browse/FLINK-22869
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.12.4, 1.13.2
>Reporter: Xintong Song
>Priority: Critical
> Labels: auto-deprioritized-critical, test-stability
> Fix For: 1.12.6, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18652=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729=27324
> {code}
> Jun 03 23:51:30 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 227.425 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase
> Jun 03 23:51:30 [ERROR]
> testReading(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 194.931 s <<< ERROR!
> Jun 03 23:51:30 org.junit.runners.model.TestTimedOutException: test timed out
> after 12 milliseconds
> Jun 03 23:51:30 at java.lang.Object.wait(Native Method)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1252)
> Jun 03 23:51:30 at java.lang.Thread.join(Thread.java:1326)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> Jun 03 23:51:30 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.KafkaContainerClient.createTopic(KafkaContainerClient.java:71)
> Jun 03 23:51:30 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testReading(SQLClientSchemaRegistryITCase.java:102)
> Jun 03 23:51:30 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jun 03 23:51:30 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jun 03 23:51:30 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jun 03 23:51:30 at java.lang.reflect.Method.invoke(Method.java:498)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Jun 03 23:51:30 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jun 03 23:51:30 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Jun 03 23:51:30 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Jun 03 23:51:30 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jun 03 23:51:30 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-23944) PulsarSourceITCase.testTaskManagerFailure is instable
[
https://issues.apache.org/jira/browse/FLINK-23944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440235#comment-17440235
]
Till Rohrmann commented on FLINK-23944:
---
cc [~arvid]
> PulsarSourceITCase.testTaskManagerFailure is instable
> -
>
> Key: FLINK-23944
> URL: https://issues.apache.org/jira/browse/FLINK-23944
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Dian Fu
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> [https://dev.azure.com/dianfu/Flink/_build/results?buildId=430=logs=f3dc9b18-b77a-55c1-591e-264c46fe44d1=2d3cd81e-1c37-5c31-0ee4-f5d5cdb9324d]
> It's from my personal azure pipeline, however, I'm pretty sure that I have
> not touched any code related to this.
> {code:java}
> Aug 24 10:44:13 [ERROR] testTaskManagerFailure{TestEnvironment,
> ExternalContext, ClusterControllable}[1] Time elapsed: 258.397 s <<< FAILURE!
> Aug 24 10:44:13 java.lang.AssertionError: Aug 24 10:44:13 Aug 24 10:44:13
> Expected: Records consumed by Flink should be identical to test data and
> preserve the order in split Aug 24 10:44:13 but: Mismatched record at
> position 7: Expected '0W6SzacX7MNL4xLL3BZ8C3ljho4iCydbvxIl' but was
> 'wVi5JaJpNvgkDEOBRC775qHgw0LyRW2HBxwLmfONeEmr' Aug 24 10:44:13 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) Aug 24 10:44:13
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) Aug 24
> 10:44:13 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testTaskManagerFailure(SourceTestSuiteBase.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19052) Performance issue with PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-19052:
---
Priority: Not a Priority (was: Minor)
> Performance issue with PojoSerializer
> -
>
> Key: FLINK-19052
> URL: https://issues.apache.org/jira/browse/FLINK-19052
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.11.1
> Environment: Flink 1.12 master on 26.08.2020
>Reporter: Roman Grebennikov
>Priority: Not a Priority
> Labels: auto-deprioritized-major, pull-request-available,
> stale-minor
> Attachments: image-2020-08-26-10-46-19-800.png,
> image-2020-08-26-10-49-59-400.png
>
>
> Currently PojoSerializer.createInstance() uses reflection call to create a
> class instance. As this method is called on each stream element on
> deserialization, reflection overhead can become noticeable in
> serialization-bound cases when:
> # Pojo class is small, so instantiation is noticeable.
> # The job is not having heavy CPU-bound event processing.
> See this flamegraph built for
> flink-benchmarks/SerializationFrameworkMiniBenchmarks.serializerPojo
> benchmark:
> !image-2020-08-26-10-46-19-800.png!
> This Reflection.getCallerClass method consumes a lot of CPU, mostly doing a
> security check if we allowed to do this reflective call.
>
> There is no true reason to perform this check on each deserializing event, so
> to speed things up we can just cache the constructor using MetaHandle, so
> this check will be performed only once. With this tiny fix, the
> getCallerClass is gone:
>
> !image-2020-08-26-10-49-59-400.png!
>
> The benchmark result:
> {noformat}
> serializerPojo thrpt 100 487.706 ± 30.480 ops/ms
> serializerPojo thrpt 100 569.828 ± 8.815 ops/m{noformat}
> Which is +15% to throughput.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] guoweiM commented on pull request #22: [FLINK-24722][iteration] Fixes the issues on supporting keyed stream
guoweiM commented on pull request #22: URL: https://github.com/apache/flink-ml/pull/22#issuecomment-962886684 In general I am fine with this fix. But I have a little problem —— Do you think that we should also add a test for `partitionCustom` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #17660: [FLINK-24657][runtime] Added metric of the total real size of input/o…
pnowojski commented on a change in pull request #17660:
URL: https://github.com/apache/flink/pull/17660#discussion_r744462195
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
##
@@ -433,6 +433,24 @@ public int getNumberOfQueuedBuffers() {
return 0;
}
+public long getSizeOfQueuedBuffers() {
+// re-try 3 times, if fails, return 0 for "unknown"
+for (int retry = 0; retry < 3; retry++) {
+try {
+long totalSize = 0;
+
+for (InputChannel channel : inputChannels.values()) {
+totalSize +=
channel.unsynchronizedGetSizeOfQueuedBuffers();
+}
+
+return totalSize;
+} catch (Exception ignored) {
Review comment:
log exception with debug?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -497,6 +497,12 @@ public int unsynchronizedGetNumberOfQueuedBuffers() {
return Math.max(buffers.size(), 0);
}
+@Override
+public long unsynchronizedGetSizeOfQueuedBuffers() {
+// Pretty rough approximation of real queue size.
+return Math.max(unsynchronizedGetNumberOfQueuedBuffers() * bufferSize,
0);
Review comment:
I have a feeling that in this implementation this metric doesn't really
tell us anything more compared to output queue length and desired buffer size.
Probably we should provide a more accurate estimate/metric or just drop it.
The question would be what to do with this for the input bytes length. As
this is only for local input channels, maybe that's a good enough estimation?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -535,7 +541,9 @@ private void updateStatistics(BufferConsumer buffer) {
}
private void updateStatistics(Buffer buffer) {
-totalNumberOfBytes += buffer.getSize();
+if (buffer.isBuffer()) {
+totalNumberOfBytes += buffer.getSize();
Review comment:
But why are you changing this in the first place? What's your
motivation?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferWritingResultPartition.java
##
@@ -111,6 +111,17 @@ public int getNumberOfQueuedBuffers() {
return totalBuffers;
}
+@Override
+public long getSizeOfQueuedBuffers() {
+long totalNumberOfBytes = 0;
+
+for (ResultSubpartition subpartition : subpartitions) {
+totalNumberOfBytes += Math.max(0,
subpartition.getTotalNumberOfBytes());
Review comment:
Rename `getSizeOfQueuedBuffers` to `getSizeOfQueuedBuffersUnsafe`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24757) Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client
[
https://issues.apache.org/jira/browse/FLINK-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440226#comment-17440226
]
Caizhi Weng commented on FLINK-24757:
-
It seems that we cannot solve it by just setting {{execution.attached}} to
false in SQL client when {{table.dml-sync}} is false and user is submitting an
insert job. This is because some execution environment does not support detach
execution (for example {{LocalExecutor#execute}} just throws an exception).
> Yarn application is not terminated after the job finishes when submitting a
> yarn-per-job insert job with SQL client
> ---
>
> Key: FLINK-24757
> URL: https://issues.apache.org/jira/browse/FLINK-24757
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> I've seen this problem for about three times in the user mailing thread
> (previously I suspect that the users are specifying the wrong
> {{{}execution.target{}}}) until I myself also bumped into this problem. I've
> submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
> job finishes Yarn application is not terminated.
> This is because yarn job cluster is using {{MiniDispatcher}} and it will
> directly terminate only in detached execution mode. This execution mode is
> (through some function calls) related to {{DeploymentOptions#ATTACHED}} which
> is true by default if jobs are submitted from SQL client.
> When submitting an insert job, SQL client will not wait for the job to
> finish. Instead it only reports the job id. So I think it is reasonable to
> set detached mode for every insert job.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] Airblader commented on a change in pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
Airblader commented on a change in pull request #17698: URL: https://github.com/apache/flink/pull/17698#discussion_r744457632 ## File path: flink-runtime-web/web-dashboard/src/app/pages/task-manager/log-list/task-manager-log-list.component.html ## @@ -29,14 +29,16 @@ Log Name -Size (KB) +Last Modified Time +Size (KB) - + Review comment: I don't mean what the type "should" be, but rather what the Angular Language Service will _actually_ infer. The last time I checked, Angular cannot use component generics to infer the type correctly. You can hover over the `log` in `let log` and see the type the language service will infer. If this is `any`, that's not good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24793) DefaultSchedulerLocalRecoveryITCase fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440224#comment-17440224
]
Zhu Zhu commented on FLINK-24793:
-
This IT case fails when it is testing using AdaptiveScheduler. The cause of the
exception is that execution history is missing in AdaptiveScheduler. When a
restarting happens, new execution graph will be re-generated, the attempt
number will be retained, while the prior executions are not inherited from the
previous execution graph.
Even if the problem above is solved, the case will still fail because local
recovery is not supported by AdaptiveScheduler yet (see FLINK-21450).
So for now I will annotate the tests with {{FailsWithAdaptiveScheduler}} so
that they can be skipped when testing AdaptiveScheduler.
> DefaultSchedulerLocalRecoveryITCase fails on AZP
>
>
> Key: FLINK-24793
> URL: https://issues.apache.org/jira/browse/FLINK-24793
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull}} and
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryRegion}} fails on AZP
> with:
> {code}
> Nov 04 23:01:32 java.lang.IllegalArgumentException: attempt does not exist
> Nov 04 23:01:32 at
> org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex.getPriorExecutionAttempt(ArchivedExecutionVertex.java:109)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.assertNonLocalRecoveredTasksEquals(DefaultSchedulerLocalRecoveryITCase.java:92)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryInternal(DefaultSchedulerLocalRecoveryITCase.java:80)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull(DefaultSchedulerLocalRecoveryITCase.java:65)
> Nov 04 23:01:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 04 23:01:32 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 04 23:01:32 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 04 23:01:32 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 04 23:01:32 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 04 23:01:32 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 04 23:01:32 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Nov 04 23:01:32 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Nov 04 23:01:32 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Nov 04 23:01:32 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Nov 04 23:01:32 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Nov 04
[GitHub] [flink] flinkbot edited a comment on pull request #17710: [FLINK-24793][tests] Skip DefaultSchedulerLocalRecoveryITCase when testing AdaptiveScheduler
flinkbot edited a comment on pull request #17710: URL: https://github.com/apache/flink/pull/17710#issuecomment-962872445 ## CI report: * cf7a81b798bd54529cb4ddeecad70fe226e10e26 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26111) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17710: [FLINK-24793][tests] Skip DefaultSchedulerLocalRecoveryITCase when testing AdaptiveScheduler
flinkbot commented on pull request #17710: URL: https://github.com/apache/flink/pull/17710#issuecomment-962872368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24757) Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client
[
https://issues.apache.org/jira/browse/FLINK-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440151#comment-17440151
]
Caizhi Weng edited comment on FLINK-24757 at 11/8/21, 7:13 AM:
---
This issue is caused by the mismatch of {{table.dml-sync}} (whose default value
is false) and {{execution.attached}} (whose default value is true, if submitted
from SQL client).
According to the offline discussion with [~twalthr], these two options are
orthogonal. {{execution.attached}} determines if cluster needs to wait for
client to fetch the result or just leaves by itself; {{table.dml-sync}}
determines if client will wait for and fetch that result.
However they are somewhat related. If client doesn’t want that result but the
cluster is willing to wait then it will have to wait forever; On the other hand
if client needs that result but cluster leaves by itself there will be an
exception.
Currently we don't have a good solution to the case when user specify the two
options at the same time, so we're just fixing it in SQL client submission and
improve the documentation for {{{}table.dml-sync{}}}. Let's come back to this
issue when a {{flink run}} + SQL user complains about this.
P.S.: The default value of {{execution.attached}} itself is false. However SQL
client is using some {{ProgramOptions}} thingy (see
{{DefaultContext#createExecutionConfig}}) in which command line options will be
read and if "-d" is found then it will be set to detach mode. But SQL client is
not passing any command line arguments to {{ProgramOptions}} (see
{{DefaultContext#new}}, it passes an empty string array) so it defaults to
attach mode.
was (Author: tsreaper):
This issue is caused by the mismatch of {{table.dml-sync}} (whose default value
is false) and {{execution.attached}} (whose default value is true).
According to the offline discussion with [~twalthr], these two options are
orthogonal. {{execution.attached}} determines if cluster needs to wait for
client to fetch the result or just leaves by itself; {{table.dml-sync}}
determines if client will wait for and fetch that result.
However they are somewhat related. If client doesn’t want that result but the
cluster is willing to wait then it will have to wait forever; On the other hand
if client needs that result but cluster leaves by itself there will be an
exception.
Currently we don't have a good solution to the case when user specify the two
options at the same time, so we're just fixing it in SQL client submission and
improve the documentation for {{table.dml-sync}}. Let's come back to this issue
when a {{flink run}} + SQL user complains about this.
> Yarn application is not terminated after the job finishes when submitting a
> yarn-per-job insert job with SQL client
> ---
>
> Key: FLINK-24757
> URL: https://issues.apache.org/jira/browse/FLINK-24757
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> I've seen this problem for about three times in the user mailing thread
> (previously I suspect that the users are specifying the wrong
> {{{}execution.target{}}}) until I myself also bumped into this problem. I've
> submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
> job finishes Yarn application is not terminated.
> This is because yarn job cluster is using {{MiniDispatcher}} and it will
> directly terminate only in detached execution mode. This execution mode is
> (through some function calls) related to {{DeploymentOptions#ATTACHED}} which
> is true by default if jobs are submitted from SQL client.
> When submitting an insert job, SQL client will not wait for the job to
> finish. Instead it only reports the job id. So I think it is reasonable to
> set detached mode for every insert job.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zhuzhurk opened a new pull request #17710: [FLINK-24793][tests] Skip DefaultSchedulerLocalRecoveryITCase when testing AdaptiveScheduler
zhuzhurk opened a new pull request #17710: URL: https://github.com/apache/flink/pull/17710 ## What is the purpose of the change DefaultSchedulerLocalRecoveryITCase does not work with AdaptiveScheduler before FLINK-21450 is done. So we skip these tests when testing AdaptiveScheduler. ## Verifying this change This change is already covered by existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24793) DefaultSchedulerLocalRecoveryITCase fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24793:
---
Labels: pull-request-available test-stability (was: test-stability)
> DefaultSchedulerLocalRecoveryITCase fails on AZP
>
>
> Key: FLINK-24793
> URL: https://issues.apache.org/jira/browse/FLINK-24793
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Zhu Zhu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull}} and
> {{DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryRegion}} fails on AZP
> with:
> {code}
> Nov 04 23:01:32 java.lang.IllegalArgumentException: attempt does not exist
> Nov 04 23:01:32 at
> org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex.getPriorExecutionAttempt(ArchivedExecutionVertex.java:109)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.assertNonLocalRecoveredTasksEquals(DefaultSchedulerLocalRecoveryITCase.java:92)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryInternal(DefaultSchedulerLocalRecoveryITCase.java:80)
> Nov 04 23:01:32 at
> org.apache.flink.test.runtime.DefaultSchedulerLocalRecoveryITCase.testLocalRecoveryFull(DefaultSchedulerLocalRecoveryITCase.java:65)
> Nov 04 23:01:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 04 23:01:32 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 04 23:01:32 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 04 23:01:32 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 04 23:01:32 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 04 23:01:32 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 04 23:01:32 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 04 23:01:32 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Nov 04 23:01:32 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Nov 04 23:01:32 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Nov 04 23:01:32 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Nov 04 23:01:32 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Nov 04 23:01:32 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Nov 04 23:01:32 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Nov 04 23:01:32 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Nov 04 23:01:32 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Nov 04 23:01:32 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Nov 04 23:01:32 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Nov 04 23:01:32 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Nov 04 23:01:32 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Nov 04 23:01:32 at
>
[jira] [Updated] (FLINK-24757) Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client
[
https://issues.apache.org/jira/browse/FLINK-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng updated FLINK-24757:
Description:
I've seen this problem for about three times in the user mailing thread
(previously I suspect that the users are specifying the wrong
{{{}execution.target{}}}) until I myself also bumped into this problem. I've
submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
job finishes Yarn application is not terminated.
This is because yarn job cluster is using {{MiniDispatcher}} and it will
directly terminate only in detached execution mode. This execution mode is
(through some function calls) related to {{DeploymentOptions#ATTACHED}} which
is true by default if jobs are submitted from SQL client.
When submitting an insert job, SQL client will not wait for the job to finish.
Instead it only reports the job id. So I think it is reasonable to set detached
mode for every insert job.
was:
I've seen this problem for about three times in the user mailing thread
(previously I suspect that the users are specifying the wrong
{{execution.target}}) until I myself also bumped into this problem. I've
submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
job finishes Yarn application is not terminated.
This is because yarn job cluster is using {{MiniDispatcher}} and it will
directly terminate only in detached execution mode. This execution mode is
(through some function calls) related to {{DeploymentOptions#ATTACHED}} which
is true by default.
When submitting an insert job, SQL client will not wait for the job to finish.
Instead it only reports the job id. So I think it is reasonable to set detached
mode for every insert job.
> Yarn application is not terminated after the job finishes when submitting a
> yarn-per-job insert job with SQL client
> ---
>
> Key: FLINK-24757
> URL: https://issues.apache.org/jira/browse/FLINK-24757
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> I've seen this problem for about three times in the user mailing thread
> (previously I suspect that the users are specifying the wrong
> {{{}execution.target{}}}) until I myself also bumped into this problem. I've
> submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
> job finishes Yarn application is not terminated.
> This is because yarn job cluster is using {{MiniDispatcher}} and it will
> directly terminate only in detached execution mode. This execution mode is
> (through some function calls) related to {{DeploymentOptions#ATTACHED}} which
> is true by default if jobs are submitted from SQL client.
> When submitting an insert job, SQL client will not wait for the job to
> finish. Instead it only reports the job id. So I think it is reasonable to
> set detached mode for every insert job.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26106) * 2b217db677c48effdbc3609657c7bbbdc46bf81a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26110) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26106) * 2b217db677c48effdbc3609657c7bbbdc46bf81a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17672: [FLINK-24761][table] Fix PartitionPruner code gen compile fail
flinkbot edited a comment on pull request #17672: URL: https://github.com/apache/flink/pull/17672#issuecomment-960546911 ## CI report: * a3b173ffd738f9c5728e92a6ee2cda4e2884228d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26102) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #17695: [FLINK-24798][cli] Bump commons-cli to v1.5.0
MartijnVisser commented on pull request #17695: URL: https://github.com/apache/flink/pull/17695#issuecomment-962866327 `commons-cli` is bundled with `org.apache.hbase.thirdparty:hbase-shaded-miscellaneous`. Flink is getting that dependency in via `org.apache.hbase:hbase-client:jar` version 2.2.1 according to the dependency tree. ``` [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ flink-sql-connector-hbase-2.2 --- [INFO] org.apache.flink:flink-sql-connector-hbase-2.2:jar:1.15-SNAPSHOT [INFO] +- org.apache.flink:flink-connector-hbase-2.2:jar:1.15-SNAPSHOT:compile [INFO] | +- org.apache.flink:flink-connector-hbase-base:jar:1.15-SNAPSHOT:compile [INFO] | +- org.apache.hbase:hbase-client:jar:2.2.3:compile [INFO] | | +- org.apache.hbase.thirdparty:hbase-shaded-protobuf:jar:2.2.1:compile [INFO] | | +- org.apache.hbase:hbase-common:jar:2.2.3:compile [INFO] | | | \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] | | +- org.apache.hbase:hbase-hadoop-compat:jar:2.2.3:compile [INFO] | | | \- org.apache.hbase:hbase-metrics-api:jar:2.2.3:compile [INFO] | | +- org.apache.hbase:hbase-hadoop2-compat:jar:2.2.3:compile [INFO] | | | \- org.apache.hbase:hbase-metrics:jar:2.2.3:compile [INFO] | | +- org.apache.hbase:hbase-protocol-shaded:jar:2.2.3:compile [INFO] | | +- org.apache.hbase:hbase-protocol:jar:2.2.3:compile [INFO] | | +- commons-codec:commons-codec:jar:1.15:compile [INFO] | | +- commons-io:commons-io:jar:2.11.0:compile [INFO] | | +- org.apache.commons:commons-lang3:jar:3.3.2:compile [INFO] | | +- org.apache.hbase.thirdparty:hbase-shaded-miscellaneous:jar:2.2.1:compile [INFO] | | | \- com.google.errorprone:error_prone_annotations:jar:2.3.3:compile [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | | +- org.apache.hbase.thirdparty:hbase-shaded-netty:jar:2.2.1:compile [INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.14:compile [INFO] | | | \- com.github.spotbugs:spotbugs-annotations:jar:3.1.9:compile [INFO] | | +- org.apache.htrace:htrace-core4:jar:4.2.0-incubating:compile [INFO] | | +- org.jruby.jcodings:jcodings:jar:1.0.18:compile [INFO] | | +- org.jruby.joni:joni:jar:2.1.11:compile [INFO] | | +- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile [INFO] | | +- org.apache.commons:commons-crypto:jar:1.0.0:compile [INFO] | | \- org.apache.yetus:audience-annotations:jar:0.5.0:compile [INFO] | \- io.netty:netty-all:jar:4.1.46.Final:compile [INFO] +- org.slf4j:slf4j-api:jar:1.7.15:provided [INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:provided [INFO] +- org.apache.flink:flink-test-utils-junit:jar:1.15-SNAPSHOT:test [INFO] +- org.apache.flink:flink-shaded-force-shading:jar:14.0:compile [INFO] +- org.junit.jupiter:junit-jupiter:jar:5.7.2:test [INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.7.2:test [INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test [INFO] | | \- org.junit.platform:junit-platform-commons:jar:1.7.2:test [INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.7.2:test [INFO] | \- org.junit.jupiter:junit-jupiter-engine:jar:5.7.2:test [INFO] +- org.junit.vintage:junit-vintage-engine:jar:5.7.2:test [INFO] | +- org.apiguardian:apiguardian-api:jar:1.1.0:test [INFO] | +- org.junit.platform:junit-platform-engine:jar:1.7.2:test [INFO] | \- junit:junit:jar:4.13.2:test [INFO] +- org.mockito:mockito-core:jar:2.21.0:test [INFO] | +- net.bytebuddy:byte-buddy:jar:1.8.15:test [INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.8.15:test [INFO] | \- org.objenesis:objenesis:jar:2.1:test [INFO] +- org.powermock:powermock-module-junit4:jar:2.0.4:test [INFO] | +- org.powermock:powermock-module-junit4-common:jar:2.0.4:test [INFO] | | +- org.powermock:powermock-reflect:jar:2.0.4:test [INFO] | | \- org.powermock:powermock-core:jar:2.0.4:test [INFO] | | \- org.javassist:javassist:jar:3.24.0-GA:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test [INFO] +- org.powermock:powermock-api-mockito2:jar:2.0.4:test [INFO] | \- org.powermock:powermock-api-support:jar:2.0.4:test [INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test [INFO] +- org.testcontainers:junit-jupiter:jar:1.16.0:test [INFO] | \- org.testcontainers:testcontainers:jar:1.16.0:test [INFO] | +- org.apache.commons:commons-compress:jar:1.21:test [INFO] | +- org.rnorth.duct-tape:duct-tape:jar:1.0.8:test [INFO] | +- com.github.docker-java:docker-java-api:jar:3.2.11:test [INFO] | | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:test [INFO] | \- com.github.docker-java:docker-java-transport-zerodep:jar:3.2.11:test [INFO] |+- com.github.docker-java:docker-java-transport:jar:3.2.11:test [INFO] |\- net.java.dev.jna:jna:jar:5.8.0:test [INFO] +-
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002) * 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107) * b51f758aa256fbc88737f0e12a53e5ed82cd067f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26109) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lincoln-lil commented on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
lincoln-lil commented on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-962865148 @JingsongLi agree with you putting these logic together. I've updated the pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26106) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002) * 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107) * b51f758aa256fbc88737f0e12a53e5ed82cd067f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17709: [FLINK-24806][connectors/rabbitmq] Fix container startup in RMQSourceITCase
flinkbot edited a comment on pull request #17709: URL: https://github.com/apache/flink/pull/17709#issuecomment-962748984 ## CI report: * ebf5af326bc99a65a14d479e4099616d288288f1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26099) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17637: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot edited a comment on pull request #17637: URL: https://github.com/apache/flink/pull/17637#issuecomment-957122792 ## CI report: * 6f0727e525cb34f2a5afbe3207dac73e177ddb33 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26100) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation
ming li created FLINK-24815:
---
Summary: Reduce the cpu cost of calculating stateSize during state
allocation
Key: FLINK-24815
URL: https://issues.apache.org/jira/browse/FLINK-24815
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing
Reporter: ming li
When the task failover, we will reassign the state for each subtask and create
a new {{OperatorSubtaskState}} object. At this time, the {{stateSize}} field in
the {{OperatorSubtaskState}} will be recalculated. When using incremental
{{{}Checkpoint{}}}, this field needs to traverse all shared states and then
accumulate the size of the state.
Taking a job with 2000 parallelism and 100 share state for each task as an
example, it needs to traverse 2000 * 100 = 20w times. At this time, the cpu of
the JM scheduling thread will be full.
I think we can try to provide a construction method with {{stateSize}} for
{{OperatorSubtaskState}} or delay the calculation of {{{}stateSize{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
flinkbot edited a comment on pull request #17698: URL: https://github.com/apache/flink/pull/17698#issuecomment-961944542 ## CI report: * 90afa69f4588fb080119395484bf7d80d99fe174 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26075) * 849c90dce7256e26e86519b328d10341b0bfceef Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26108) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
flinkbot edited a comment on pull request #17698: URL: https://github.com/apache/flink/pull/17698#issuecomment-961944542 ## CI report: * 90afa69f4588fb080119395484bf7d80d99fe174 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26075) * 849c90dce7256e26e86519b328d10341b0bfceef UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17708: [FLINK-23696][connectors/rabbitmq] Fix RMQSourceTest.testRedeliveredSessionIDsAck
flinkbot edited a comment on pull request #17708: URL: https://github.com/apache/flink/pull/17708#issuecomment-962715413 ## CI report: * 1fe63c9384f63908b2f0024172d8e6753bdbb84f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26097) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ysymi edited a comment on pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
ysymi edited a comment on pull request #17698: URL: https://github.com/apache/flink/pull/17698#issuecomment-962836982 I will rebase on latest master (commit: fc4f2556) to pass the CI check in Azure. The reason is the master I checkout from (commit: db04af41 ) always failed in `org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicTableFactoryTest#testTableSourceWithKeyValueAndMetadata`, because the valueDecodingFormat.producedDataType was different between two KafkaSource(one has `not null` but another one not have). And looks like this problem has been solved by commit 2a582f3. So I will rebase on laster master to pass the CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ysymi commented on pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
ysymi commented on pull request #17698: URL: https://github.com/apache/flink/pull/17698#issuecomment-962836982 I will rebase on latest master (commit: fc4f2556) to pass the CI check in Azure. The reason is the master I checkout from (commit: db04af41 ) always failed in `org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicTableFactoryTest#testTableSourceWithKeyValueAndMetadata`, because the valueDecodingFormat.producedDataType was different between two KafkaSource(one has `not null` but another one not have). And looks like this problem has been solved by commit 2a582f3. So I will rebase on laster master to pass the CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #10: [FLINK-24354][FLIP-174] Improve the WithParams interface
lindong28 commented on a change in pull request #10:
URL: https://github.com/apache/flink-ml/pull/10#discussion_r744420186
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/param/Param.java
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.param;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.util.ReadWriteUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+
+/**
+ * Definition of a parameter, including name, class, description, default
value and the validator.
+ *
+ * @param The class type of the parameter value.
+ */
+@PublicEvolving
+public class Param implements Serializable {
+private static final long serialVersionUID = 4396556083935765299L;
+
+public final String name;
+public final Class clazz;
+public final String description;
+public final T defaultValue;
+public final ParamValidator validator;
+
+public Param(
+String name,
+Class clazz,
+String description,
+T defaultValue,
+ParamValidator validator) {
+this.name = name;
+this.clazz = clazz;
+this.description = description;
+this.defaultValue = defaultValue;
+this.validator = validator;
+
+if (defaultValue != null && !validator.validate(defaultValue)) {
Review comment:
Discussed offline. We agreed to keep the check as it is. The reason is
that we want to allow users not to set the default value (e.g. defaultValue =
null) in the parameter definition and still be able to specify a validator
which requires the parameter value to be not-null.
Note that the check is useful here in case users specify a not-null default
value that does not meet the validator requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
JingsongLi commented on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-962830715 Thanks @lincoln-lil Note `SinkUpsertMaterializer` requires update before. We should not separate the inference of change log mode from the sink upsert materializer, which will lead to bugs. We should put both the inference and sink upsert mode in one place. I think we should introduce something like `RankProcessStrategy` of rank in `FlinkChangelogModeInferenceProgram`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002) * 881206c906525fb7f4d1f9391bc3b196838d62fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26107) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * de953e065ada40720bd5118aa8fb43842b27733b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25002) * 881206c906525fb7f4d1f9391bc3b196838d62fa UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * 4115ff9fb202b7fbb138fed2c988ab1189345bcc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26082) * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26106) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * 4115ff9fb202b7fbb138fed2c988ab1189345bcc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26082) * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) * 8c7a80e8f57029838eb0be04ae27fa0d64001ff4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24814) Cannot update accumulators for job
mo.wu created FLINK-24814: - Summary: Cannot update accumulators for job Key: FLINK-24814 URL: https://issues.apache.org/jira/browse/FLINK-24814 Project: Flink Issue Type: Bug Components: Table SQL / Runtime Affects Versions: 1.13.2 Environment: hudi 0.10.0-SNAPSHOT + flink 1.13.2 Reporter: mo.wu i got a problem while flink on hudi ERROR org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Cannot update accumulators for job 6192c8f6e7821e006f64e13d90a1f2be. java.lang.NullPointerException: null at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.updateAccumulators(DefaultExecutionGraph.java:1402) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.scheduler.SchedulerBase.updateAccumulators(SchedulerBase.java:821) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.reportPayload(JobMaster.java:1313) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.reportPayload(JobMaster.java:1288) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl.receiveHeartbeat(HeartbeatManagerImpl.java:200) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.jobmaster.JobMaster.heartbeatFromTaskManager(JobMaster.java:771) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:301) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.13.2.jar:1.13.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.13.2.jar:1.13.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.13.2.jar:1.13.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.13.2.jar:1.13.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.13.2.jar:1.13.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.13.2.jar:1.13.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.13.2.jar:1.13.2] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] cshuo commented on a change in pull request #17634: [FLINK-24691][table-common] Fix precision for sum on decimal types
cshuo commented on a change in pull request #17634:
URL: https://github.com/apache/flink/pull/17634#discussion_r744380537
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java
##
@@ -759,6 +760,24 @@
explicit(DataTypes.STRING()
.build();
+/**
+ * Special "+" operator used internally by {@code SumAggFunction} to
implement SUM aggregation
+ * on a Decimal type. Uses the {@link
LogicalTypeMerging#findSumAggType(LogicalType)} to avoid
+ * the normal {@link #PLUS} override the special calculation for precision
and scale needed by
+ * SUM.
+ */
+public static final BuiltInFunctionDefinition PLUS_AGG =
+BuiltInFunctionDefinition.newBuilder()
+.name("PLUS_AGG")
Review comment:
`PLUS_AGG` looks like an aggregate function..Maybe
`AGG_PLUS`、`PLUS_FOR_AGG`..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24809) Check & possible fix decimal precision and scale for all Aggregate functions
[ https://issues.apache.org/jira/browse/FLINK-24809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440170#comment-17440170 ] Shuo Cheng commented on FLINK-24809: `SumWithRetractAggFunction` as well. > Check & possible fix decimal precision and scale for all Aggregate functions > > > Key: FLINK-24809 > URL: https://issues.apache.org/jira/browse/FLINK-24809 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Marios Trivyzas >Priority: Major > > Similar to FLINK-24691, check the behaviour of the rest of Aggregate > functions like *Sum0AggFunction, AvgAggFunction,* etc. regarding the > precision/scale of the resulting decimal. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] ysymi commented on pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
ysymi commented on pull request #17698: URL: https://github.com/apache/flink/pull/17698#issuecomment-962787085 @flinkbot re-run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ysymi commented on a change in pull request #17698: [FLINK-24689][runtime-web] Add log's last modify time in log list view
ysymi commented on a change in pull request #17698: URL: https://github.com/apache/flink/pull/17698#discussion_r744387331 ## File path: flink-runtime-web/web-dashboard/src/app/pages/task-manager/log-list/task-manager-log-list.component.html ## @@ -29,14 +29,16 @@ Log Name -Size (KB) +Last Modified Time +Size (KB) - + Review comment: Thanks for review code firstly. I think table.data's type here is `TaskManagerLogItemInterface[]` and table.data's type in `job-manager-log-list.component.html` is `JobManagerLogItemInterface[]`. I read the doc of [NG_ZORRO#table](https://ng.ant.design/components/table/en#components-table-demo-basic) , and it's said that table.data's value is set by nz-table's property `[nzData]`, so I passed `listOfLog` to `nzData` which is defined in related .ts file . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] cshuo commented on a change in pull request #17634: [FLINK-24691][table-common] Fix precision for sum on decimal types
cshuo commented on a change in pull request #17634:
URL: https://github.com/apache/flink/pull/17634#discussion_r744367549
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala
##
@@ -834,6 +834,11 @@ class ExprCodeGenerator(ctx: CodeGeneratorContext,
nullableInput: Boolean)
case BuiltInFunctionDefinitions.JSON_STRING =>
new JsonStringCallGen(call).generate(ctx, operands, resultType)
+ case BuiltInFunctionDefinitions.PLUS_AGG =>
+val left = operands.head
+val right = operands(1)
Review comment:
add some check for operands? e.g., requireDecimal(..)
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java
##
@@ -759,6 +760,24 @@
explicit(DataTypes.STRING()
.build();
+/**
+ * Special "+" operator used internally by {@code SumAggFunction} to
implement SUM aggregation
+ * on a Decimal type. Uses the {@link
LogicalTypeMerging#findSumAggType(LogicalType)} to avoid
+ * the normal {@link #PLUS} override the special calculation for precision
and scale needed by
+ * SUM.
+ */
+public static final BuiltInFunctionDefinition PLUS_AGG =
+BuiltInFunctionDefinition.newBuilder()
+.name("PLUS_AGG")
Review comment:
`PLUS_AGG` looks like is an aggregate function..Maybe
`AGG_PLUS`、`PLUS_FOR_AGG`..
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/aggfunctions/SumAggFunction.java
##
@@ -149,15 +161,24 @@ public DataType getResultType() {
/** Built-in Decimal Sum aggregate function. */
public static class DecimalSumAggFunction extends SumAggFunction {
private DecimalType decimalType;
+private DataType returnType;
public DecimalSumAggFunction(DecimalType decimalType) {
this.decimalType = decimalType;
}
@Override
public DataType getResultType() {
-DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType);
-return DataTypes.DECIMAL(sumType.getPrecision(),
sumType.getScale());
+if (returnType == null) {
+DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType);
+returnType = DataTypes.DECIMAL(sumType.getPrecision(),
sumType.getScale());
+}
+return returnType;
+}
+
Review comment:
`@Override`
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/aggfunctions/SumAggFunction.java
##
@@ -66,7 +68,10 @@ public int operandCount() {
ifThenElse(
Review comment:
`adjustSumType` is not needed any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24279) Support withBroadcast with DataStream API in Flink ML Library
[
https://issues.apache.org/jira/browse/FLINK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440162#comment-17440162
]
Yun Gao commented on FLINK-24279:
-
Fix on master via 8269bd9fdf3e5744b2d635697db5c705b9e598f5
> Support withBroadcast with DataStream API in Flink ML Library
> --
>
> Key: FLINK-24279
> URL: https://issues.apache.org/jira/browse/FLINK-24279
> Project: Flink
> Issue Type: New Feature
> Components: Library / Machine Learning
>Reporter: Zhipeng Zhang
>Priority: Major
> Labels: pull-request-available
>
> When doing machine learning using DataStream, we found that DataStream lacks
> withBroadcast() function, which could be useful in machine learning.
>
> A DataSet-based demo is like:
> {code:java}
> DataSet d1 = ...;
> DataSet d2 = ...;
> d1.map(new RichMapFunction () {
> @Override
> public Object map(Object aLong) throws Exception{
> List elements = getRuntimeContext().getBroadcastVariable("d2");
> ...;
>}
> }).withBroadcastSet(d2, "d2");
> {code}
>
> The withBroadcast() function incurs priority-base data-consuming. For example
> in the above code snippet, we cannot consume any element from d1 before we
> consumed all of elements in d2.
>
> Thus when supporting withBroadcast() in DataStream, we also need
> priority-base data-consuming. This could probably lead to deadlock and
> DataStream does not provide a solution for deadlock.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-24279) Support withBroadcast with DataStream API in Flink ML Library
[
https://issues.apache.org/jira/browse/FLINK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao closed FLINK-24279.
---
Fix Version/s: 0.1.0
Assignee: Zhipeng Zhang
Resolution: Fixed
> Support withBroadcast with DataStream API in Flink ML Library
> --
>
> Key: FLINK-24279
> URL: https://issues.apache.org/jira/browse/FLINK-24279
> Project: Flink
> Issue Type: New Feature
> Components: Library / Machine Learning
>Reporter: Zhipeng Zhang
>Assignee: Zhipeng Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.1.0
>
>
> When doing machine learning using DataStream, we found that DataStream lacks
> withBroadcast() function, which could be useful in machine learning.
>
> A DataSet-based demo is like:
> {code:java}
> DataSet d1 = ...;
> DataSet d2 = ...;
> d1.map(new RichMapFunction () {
> @Override
> public Object map(Object aLong) throws Exception{
> List elements = getRuntimeContext().getBroadcastVariable("d2");
> ...;
>}
> }).withBroadcastSet(d2, "d2");
> {code}
>
> The withBroadcast() function incurs priority-base data-consuming. For example
> in the above code snippet, we cannot consume any element from d1 before we
> consumed all of elements in d2.
>
> Thus when supporting withBroadcast() in DataStream, we also need
> priority-base data-consuming. This could probably lead to deadlock and
> DataStream does not provide a solution for deadlock.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] gaoyunhaii closed pull request #18: [FLINK-24279] Support withBroadcast in DataStream by caching in static variables
gaoyunhaii closed pull request #18: URL: https://github.com/apache/flink-ml/pull/18 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24783) Improve monitoring experience and usability of state backend
[ https://issues.apache.org/jira/browse/FLINK-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440160#comment-17440160 ] Yun Tang edited comment on FLINK-24783 at 11/8/21, 3:13 AM: [~sjwiesman] [~nkruber], I think it would be helpful to add such configurable option to forward RocksDB logs, and you could create a sub-task under this ticket. was (Author: yunta): [~sjwiesman][~nkruber], I think it would be helpful to add such configurable option to forward RocksDB logs, and you could create a sub-task under this ticket. > Improve monitoring experience and usability of state backend > - > > Key: FLINK-24783 > URL: https://issues.apache.org/jira/browse/FLINK-24783 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > > This ticket targets for improving the monitoring experiences and usability > for HashMap and EmbededRocksDB state backends. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24783) Improve monitoring experience and usability of state backend
[ https://issues.apache.org/jira/browse/FLINK-24783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440160#comment-17440160 ] Yun Tang commented on FLINK-24783: -- [~sjwiesman][~nkruber], I think it would be helpful to add such configurable option to forward RocksDB logs, and you could create a sub-task under this ticket. > Improve monitoring experience and usability of state backend > - > > Key: FLINK-24783 > URL: https://issues.apache.org/jira/browse/FLINK-24783 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > > This ticket targets for improving the monitoring experiences and usability > for HashMap and EmbededRocksDB state backends. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] unigeorge commented on pull request #17585: [hotfix][docs] Modify wrong variable names in doc demo code
unigeorge commented on pull request #17585: URL: https://github.com/apache/flink/pull/17585#issuecomment-962769793 @dianfu Can you review it for me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20374) Wrong result when shuffling changelog stream on non-primary-key columns
[
https://issues.apache.org/jira/browse/FLINK-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440157#comment-17440157
]
Leonard Xu commented on FLINK-20374:
[~lzljs3620320] Do you have plan to cp this fix to release-1.12 ?
> Wrong result when shuffling changelog stream on non-primary-key columns
> ---
>
> Key: FLINK-20374
> URL: https://issues.apache.org/jira/browse/FLINK-20374
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Jark Wu
>Assignee: Jingsong Lee
>Priority: Critical
> Labels: auto-deprioritized-critical
> Fix For: 1.14.0, 1.13.3
>
>
> This is reported from user-zh ML:
> http://apache-flink.147419.n8.nabble.com/flink-1-11-2-cdc-cdc-sql-sink-save-point-job-sink-td8593.html
> {code:sql}
> CREATE TABLE test (
> `id` INT,
> `name` VARCHAR(255),
> `time` TIMESTAMP(3),
> `status` INT,
> PRIMARY KEY(id) NOT ENFORCED
> ) WITH (
> 'connector' = 'mysql-cdc',
> 'hostname' = 'localhost',
> 'port' = '3306',
> 'username' = 'root',
> 'password' = '1',
> 'database-name' = 'ai_audio_lyric_task',
> 'table-name' = 'test'
> )
> CREATE TABLE status (
> `id` INT,
> `name` VARCHAR(255),
> PRIMARY KEY(id) NOT ENFORCED
> ) WITH (
> 'connector' = 'mysql-cdc',
> 'hostname' = 'localhost',
> 'port' = '3306',
> 'username' = 'root',
> 'password' = '1',
> 'database-name' = 'ai_audio_lyric_task',
> 'table-name' = 'status'
> );
> -- output
> CREATE TABLE test_status (
> `id` INT,
> `name` VARCHAR(255),
> `time` TIMESTAMP(3),
> `status` INT,
> `status_name` VARCHAR(255)
> PRIMARY KEY(id) NOT ENFORCED
> ) WITH (
> 'connector' = 'elasticsearch-7',
> 'hosts' = 'xxx',
> 'index' = 'xxx',
> 'username' = 'xxx',
> 'password' = 'xxx',
> 'sink.bulk-flush.backoff.max-retries' = '10',
> 'sink.bulk-flush.backoff.strategy' = 'CONSTANT',
> 'sink.bulk-flush.max-actions' = '5000',
> 'sink.bulk-flush.max-size' = '10mb',
> 'sink.bulk-flush.interval' = '1s'
> );
> INSERT into test_status
> SELECT t.*, s.name
> FROM test AS t
> LEFT JOIN status AS s ON t.status = s.id;
> {code}
> Data in mysql table:
> {code}
> test:
> 0, name0, 2020-07-06 00:00:00 , 0
> 1, name1, 2020-07-06 00:00:00 , 1
> 2, name2, 2020-07-06 00:00:00 , 1
> .
> status
> 0, status0
> 1, status1
> 2, status2
> .
> {code}
> Operations:
> 1. start job with paralleslim=40, result in test_status sink is correct:
> {code}
> 0, name0, 2020-07-06 00:00:00 , 0, status0
> 1, name1, 2020-07-06 00:00:00 , 1, status1
> 2, name2, 2020-07-06 00:00:00 , 1, status1
> {code}
> 2. Update {{status}} of {{id=2}} record in table {{test}} from {{1}} to {{2}}.
> 3. Result is not correct because the {{id=2}} record is missing in the
> result.
> The reason is that it shuffles the changelog {{test}} on {{status}} column
> which is not the primary key. Therefore, the ordering can't be guaranteed,
> and the result is wrong.
> The {{-U[2, name2, 2020-07-06 00:00:00 , 1]}} and {{+U[2, name2, 2020-07-06
> 00:00:00 , 2]}} will possible be shuffled to different join task, so the
> order of joined results is not guaranteed when they arrive to the sink task.
> It is possbile {{+U[2, name2, 2020-07-06 00:00:00 , status2]}} arrives
> first, and then {{-U[2, name2, 2020-07-06 00:00:00 , status1]}} , then the
> {{id=2}} record is missing in Elasticsearch.
> It seems that we need a changelog ordering mechanism in the planner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-24475) Remove no longer used NestedMap* classes
[
https://issues.apache.org/jira/browse/FLINK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang resolved FLINK-24475.
--
Resolution: Fixed
Merged in master: fc4f255644a64bb556b0dcefb165a9c772164c5b
> Remove no longer used NestedMap* classes
>
>
> Key: FLINK-24475
> URL: https://issues.apache.org/jira/browse/FLINK-24475
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Zakelly Lan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> After FLINK-21935 all of the {{NestedMapsStateTable}} classes are no longer
> used in the production code. They are still however being used in benchmarks
> in some tests. Benchmarks/tests should be migrated to {{CopyOnWrite}}
> versions while the {{NestedMaps}} classes should be removed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] Myasuka closed pull request #17650: [FLINK-24475] Remove no longer used NestedMap* classes
Myasuka closed pull request #17650: URL: https://github.com/apache/flink/pull/17650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24813) Improve ImplicitTypeConversionITCase
[ https://issues.apache.org/jira/browse/FLINK-24813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-24813: -- Assignee: xuyang > Improve ImplicitTypeConversionITCase > > > Key: FLINK-24813 > URL: https://issues.apache.org/jira/browse/FLINK-24813 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.15.0, 1.14.1, 1.13.4 >Reporter: godfrey he >Assignee: xuyang >Priority: Minor > Fix For: 1.15.0, 1.14.1, 1.13.4 > > > Remove the legacy code and merge them into the current cast testing -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24813) Improve ImplicitTypeConversionITCase
godfrey he created FLINK-24813: -- Summary: Improve ImplicitTypeConversionITCase Key: FLINK-24813 URL: https://issues.apache.org/jira/browse/FLINK-24813 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.15.0, 1.14.1, 1.13.4 Reporter: godfrey he Fix For: 1.15.0, 1.14.1, 1.13.4 Remove the legacy code and merge them into the current cast testing -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] godfreyhe edited a comment on pull request #17444: [FLINK-24492][table-planner]incorrect implicit type conversion between numeric and (var)char
godfreyhe edited a comment on pull request #17444: URL: https://github.com/apache/flink/pull/17444#issuecomment-962765849 ok, I will create an issue to improve the tests, FLINK-24813 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15352) develop MySQLCatalog to connect Flink with MySQL tables and ecosystem
[ https://issues.apache.org/jira/browse/FLINK-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440155#comment-17440155 ] Yuepeng Pan commented on FLINK-15352: - [~airblader] Thank you so much for the review. I made some change. Could you help me to check it again? > develop MySQLCatalog to connect Flink with MySQL tables and ecosystem > -- > > Key: FLINK-15352 > URL: https://issues.apache.org/jira/browse/FLINK-15352 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: Bowen Li >Assignee: Yuepeng Pan >Priority: Minor > Labels: pull-request-available > Attachments: research-results.tar.gz > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] godfreyhe commented on pull request #17444: [FLINK-24492][table-planner]incorrect implicit type conversion between numeric and (var)char
godfreyhe commented on pull request #17444: URL: https://github.com/apache/flink/pull/17444#issuecomment-962765849 ok, I will create a issue to improve the tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #17637: [FLINK-24708][planner] Fix wrong results of the IN operator
godfreyhe commented on a change in pull request #17637:
URL: https://github.com/apache/flink/pull/17637#discussion_r744369780
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/plan/rules/logical/ConvertToNotInOrInRuleTest.scala
##
@@ -169,4 +169,32 @@ class ConvertToNotInOrInRuleTest extends TableTestBase {
"SELECT * FROM MyTable WHERE b = 1 OR b = 2 OR (a <> 1 AND a <> 2 AND a
<> 3 " +
"AND a <> 4 AND c = 1) OR b = 3 OR b = 4 OR c = 1")
}
+
+ @Test
+ def testConvertToSearch(): Unit = {
+util.verifyRelPlan(
+ """
+|SELECT * from MyTable where e in (
+|'CTNBSmokeSensor',
+|'H388N',
+|'H389N ',
+|'GHL-IRD',
+|'JY-BF-20YN',
+|'HC809',
+|'DH-9908N-AEP',
+|'DH-9908N'
+|)
+|""".stripMargin
+)
+ }
+
+ @Test
+ def testConvertToSearch2(): Unit = {
+util.verifyRelPlan(
+ "SELECT * FROM MyTable WHERE " +
+"e = 'a' or e = 'b' or e = 'c' or e = 'd' or e = 'e' or e = 'f' or e =
NULL or e = " +
+"'HELLO " +
+"WORLD!'"
+)
Review comment:
please add some tests to cover other types (such as, int, bigint, double)
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/CalcITCase.scala
##
@@ -1419,6 +1419,76 @@ class CalcITCase extends BatchTestBase {
)
}
+ @Test
+ def testSearch(): Unit = {
+val myTableDataId = TestValuesTableFactory.registerData(
+ Seq(row("HC809"), row("H389N "))
+)
+val ddl =
+ s"""
+ |CREATE TABLE SimpleTable (
+ | content STRING
+ |) WITH (
+ | 'connector' = 'values',
+ | 'data-id' = '$myTableDataId',
+ | 'bounded' = 'true'
+ |)
+ |""".stripMargin
+tEnv.executeSql(ddl)
+val sql =
+ """
+|SELECT UPPER(content) from SimpleTable where UPPER(content) in (
+|'CTNBSmokeSensor',
+|'H388N',
+|'H389N ',
+|'GHL-IRD',
+|'JY-BF-20YN',
+|'HC809',
+|'DH-9908N-AEP',
+|'DH-9908N'
+|)
+|
+|""".stripMargin
+checkResult(
+ sql,
+ Seq(row("HC809"), row("H389N "))
+)
+ }
+
+ @Test
+ def testSearchWithNull(): Unit = {
Review comment:
please add a test that the first element is null
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/ConvertToNotInOrInRule.scala
##
@@ -184,6 +188,52 @@ class ConvertToNotInOrInRule
case _ => rexNodes.size >= THRESHOLD
}
}
+
+
+ /**
+ * Convert the conditions into the [[IN]] and fix the bug in
[[RexBuilder#makeIn]].
Review comment:
Is there any Calcite issue tracking this bug？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-20370) Result is wrong when sink primary key is not the same with query
[
https://issues.apache.org/jira/browse/FLINK-20370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440153#comment-17440153
]
Wenlong Lyu commented on FLINK-20370:
-
[~twalthr] thanks for the explanation.
However, I think when the sink is kafka with pk(I would assume that it is a
upsert kafka with key format provided), the target Kafka partition of a record
is decided by the key generated. event it is written at different subtask.
In such case, I think adding a key-by can help keep the order of record with
the same uid, only when the input is already partitioned by uid. Is this case
is what you want to solve?
> Result is wrong when sink primary key is not the same with query
>
>
> Key: FLINK-20370
> URL: https://issues.apache.org/jira/browse/FLINK-20370
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Jark Wu
>Assignee: lincoln lee
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Both sources are upsert-kafka which synchronizes the changes from MySQL
> tables (source_city, source_customer). The sink is another MySQL table which
> is in upsert mode with "city_name" primary key. The join key is "city_id".
> In this case, the result will be wrong when updating
> {{source_city.city_name}} column in MySQL, as the UPDATE_BEFORE is ignored
> and the old city_name is retained in the sink table.
> {code}
> Sink(table=[default_catalog.default_database.sink_kafka_count_city],
> fields=[city_name, count_customer, sum_gender], changelogMode=[NONE])
> +- Calc(select=[city_name, CAST(count_customer) AS count_customer,
> CAST(sum_gender) AS sum_gender], changelogMode=[I,UA,D])
>+- Join(joinType=[InnerJoin], where=[=(city_id, id)], select=[city_id,
> count_customer, sum_gender, id, city_name],
> leftInputSpec=[JoinKeyContainsUniqueKey],
> rightInputSpec=[JoinKeyContainsUniqueKey], changelogMode=[I,UA,D])
> :- Exchange(distribution=[hash[city_id]], changelogMode=[I,UA,D])
> : +- GlobalGroupAggregate(groupBy=[city_id], select=[city_id,
> COUNT_RETRACT(count1$0) AS count_customer, SUM_RETRACT((sum$1, count$2)) AS
> sum_gender], changelogMode=[I,UA,D])
> : +- Exchange(distribution=[hash[city_id]], changelogMode=[I])
> :+- LocalGroupAggregate(groupBy=[city_id], select=[city_id,
> COUNT_RETRACT(*) AS count1$0, SUM_RETRACT(gender) AS (sum$1, count$2)],
> changelogMode=[I])
> : +- Calc(select=[city_id, gender], changelogMode=[I,UB,UA,D])
> : +- ChangelogNormalize(key=[customer_id],
> changelogMode=[I,UB,UA,D])
> : +- Exchange(distribution=[hash[customer_id]],
> changelogMode=[UA,D])
> :+- MiniBatchAssigner(interval=[3000ms],
> mode=[ProcTime], changelogMode=[UA,D])
> : +- TableSourceScan(table=[[default_catalog,
> default_database, source_customer]], fields=[customer_id, city_id, age,
> gender, update_time], changelogMode=[UA,D])
> +- Exchange(distribution=[hash[id]], changelogMode=[I,UA,D])
> +- ChangelogNormalize(key=[id], changelogMode=[I,UA,D])
> +- Exchange(distribution=[hash[id]], changelogMode=[UA,D])
>+- MiniBatchAssigner(interval=[3000ms], mode=[ProcTime],
> changelogMode=[UA,D])
> +- TableSourceScan(table=[[default_catalog,
> default_database, source_city]], fields=[id, city_name], changelogMode=[UA,D])
> {code}
> We have suggested users to use the same key of the query as the primary key
> on sink in the documentation:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#deduplication.
> We should make this attention to be more highlight in CREATE TABLE page.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * 4115ff9fb202b7fbb138fed2c988ab1189345bcc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26082) * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26103) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17672: [FLINK-24761][table] Fix PartitionPruner code gen compile fail
flinkbot edited a comment on pull request #17672: URL: https://github.com/apache/flink/pull/17672#issuecomment-960546911 ## CI report: * 85f69400f532e6d1732fc9cd26c03cbb5ace21ad Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26023) * a3b173ffd738f9c5728e92a6ee2cda4e2884228d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26102) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17699: [FLINK-20370][table] Fix wrong results when sink primary key is not the same with query result's changelog upsert key
flinkbot edited a comment on pull request #17699: URL: https://github.com/apache/flink/pull/17699#issuecomment-961963701 ## CI report: * 4115ff9fb202b7fbb138fed2c988ab1189345bcc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26082) * bc1d328642ab4c430d4e8d0cbf977d26eb97d2e6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17672: [FLINK-24761][table] Fix PartitionPruner code gen compile fail
flinkbot edited a comment on pull request #17672: URL: https://github.com/apache/flink/pull/17672#issuecomment-960546911 ## CI report: * 85f69400f532e6d1732fc9cd26c03cbb5ace21ad Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26023) * a3b173ffd738f9c5728e92a6ee2cda4e2884228d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Tartarus0zm commented on pull request #17672: [FLINK-24761][table] Fix PartitionPruner code gen compile fail
Tartarus0zm commented on pull request #17672: URL: https://github.com/apache/flink/pull/17672#issuecomment-962761027 @tsreaper thanks for your review. I has fix the comments, please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
