[GitHub] [flink] aljoscha commented on pull request #11326: [FLINK-11427][formats] Add protobuf parquet support for StreamingFileSink
aljoscha commented on pull request #11326: URL: https://github.com/apache/flink/pull/11326#issuecomment-702031110 I think we should merge this after all. The code is very simple but if we provide it we avoid different users having to re-implement it in their own code. Plus, having the code in an example as text can go out of date and I would avoid the problem that users copy the code from the documentation and it then doesn't work with the current interfaces. Also, I think we don't need to update the `NOTICE` file because the new dependencies are optional. The change doesn't introduce any new dependencies that are not optional or in `test` scope, so it's quite minimal. I do think @JingsongLi makes a good point but I think here I think it's more important that we don't have code in the documentation that can "rot". WDYT? @gaoyunhaii If we agree to merge, could you please rebase on `master` and make sure that it still works? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rutuja2807 opened a new pull request #384: Improved Docs
rutuja2807 opened a new pull request #384: URL: https://github.com/apache/flink-web/pull/384 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19447) HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not initialized after 200000ms"
[
https://issues.apache.org/jira/browse/FLINK-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyula Fora reassigned FLINK-19447:
--
Assignee: Miklos Gergely
> HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not
> initialized after 20ms"
> ---
>
> Key: FLINK-19447
> URL: https://issues.apache.org/jira/browse/FLINK-19447
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Miklos Gergely
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=60581941-0138-53c0-39fe-86d62be5f407
> {code}
> 2020-09-28T21:52:21.2146147Z
> org.apache.flink.connector.hbase2.HBaseConnectorITCase Time elapsed: 208.382
> sec <<< ERROR!
> 2020-09-28T21:52:21.2146638Z java.io.IOException: Shutting down
> 2020-09-28T21:52:21.2147004Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
> 2020-09-28T21:52:21.2147637Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:116)
> 2020-09-28T21:52:21.2148120Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1142)
> 2020-09-28T21:52:21.2148831Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
> 2020-09-28T21:52:21.2149347Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1061)
> 2020-09-28T21:52:21.2149896Z at
> org.apache.flink.connector.hbase2.util.HBaseTestingClusterAutoStarter.setUp(HBaseTestingClusterAutoStarter.java:122)
> 2020-09-28T21:52:21.2150721Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-09-28T21:52:21.2151136Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-09-28T21:52:21.2151609Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-09-28T21:52:21.2152039Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-09-28T21:52:21.2152462Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-09-28T21:52:21.2152941Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-09-28T21:52:21.2153489Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-09-28T21:52:21.2153962Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-09-28T21:52:21.2154406Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-09-28T21:52:21.2154828Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-09-28T21:52:21.2155381Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
> 2020-09-28T21:52:21.2155864Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
> 2020-09-28T21:52:21.2156378Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-09-28T21:52:21.2156865Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> 2020-09-28T21:52:21.2157458Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> 2020-09-28T21:52:21.2157993Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> 2020-09-28T21:52:21.2158470Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> 2020-09-28T21:52:21.2158890Z Caused by: java.lang.RuntimeException: Master
> not initialized after 20ms
> 2020-09-28T21:52:21.2159350Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
> 2020-09-28T21:52:21.2159823Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
> 2020-09-28T21:52:21.2160270Z at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> 2020-09-28T21:52:21.2160800Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
> 2020-09-28T21:52:21.2161096Z ... 22 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] aljoscha commented on pull request #13473: [FLINK-19264][runtime] Add JobID to ExecutionAttemptID
aljoscha commented on pull request #13473: URL: https://github.com/apache/flink/pull/13473#issuecomment-702023979 So? Can we merge? 樂 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18662) Provide more detailed metrics why unaligned checkpoint is taking long time
[ https://issues.apache.org/jira/browse/FLINK-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205396#comment-17205396 ] Piotr Nowojski edited comment on FLINK-18662 at 10/1/20, 9:34 AM: -- I’ve found an issue with “processed data during alignment” metric. For unaligned checkpoints the metrics that would be the most interesting are: 1. persisted in-flight data in the ckeckpoint (that’s easy to do) 2. something to compare, how persisted in-flight data compare to processed data during the time in-flight data were being persisted If number of processed data in 2. is close to 1., UC do not make much sense. But the number that we are looking for here, is not “processed during alignment”, but “processed during checkpoint” (including the async phase). "processed during alignment" would be the amount of data processed between first and last received checkpoint barrier, which can be very small amount of data if UC barriers travel very quickly, while the time to persist the overtaken in-flight data can be still huge, vastly exceeding the "alignment" time. So I'm going to make this adjustment in my pr (instead of "during alignment" calculate the new metrics "during checkpoint") was (Author: pnowojski): I’ve found an issue with “processed data during alignment” metric. For unaligned checkpoints the metrics that would be the most interesting are: 1. persisted in-flight data in the ckeckpoint (that’s easy to do) 2. something to compare, how persisted in-flight data compare to processed data during the time in-flight data were being persisted If number of processed data in 2. is close to 1., UC do not make much sense. But the number that we are looking for here, is not “processed during alignment”, but “processed during checkpoint” (including the async phase). "processed during alignment" would be the amount of data processed between first and last received checkpoint barrier. So I'm going to make this adjustment in my pr (instead of "during alignment" calculate the new metrics "during checkpoint") > Provide more detailed metrics why unaligned checkpoint is taking long time > -- > > Key: FLINK-18662 > URL: https://issues.apache.org/jira/browse/FLINK-18662 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Network >Affects Versions: 1.11.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Critical > Fix For: 1.12.0 > > Attachments: Screenshot 2020-07-21 at 11.50.02.png > > > With unaligned checkpoint there can happen situation as in the attached > screenshot. > Task reports long end to end checkpoint time (~2h50min), ~0s sync time, > ~2h50min async time, ~0s start delay. It means that task received first > checkpoint barrier from one of the channels very quickly (~0s), sync part was > quick, but we do not know why async part was taking so long. It could be > because of three things: > # long operator state IO writes > # long spilling of in-flight data > # long time to receive the final checkpoint barrier from the last lagging > channel > First and second are probably indistinguishable and the difference between > them doesn't matter much for analyzing. However the last one is quite > different. It might be independent of the IO, and we are missing this > information. > Maybe we could report it as "alignment duration" and while we are at it, we > could also report amount of spilled in-flight data for unaligned checkpoints > as "alignment buffered"? > Ideally we should report it as new metrics, but that leaves a question how to > display it in the UI, with limited space available. Maybe it could be > reported as: > ||Alignment Buffered||Alignment Duration|| > |0 B (632 MB)|0ms (2h 49m 32s)| > Where the values in the parenthesis would come from unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18662) Provide more detailed metrics why unaligned checkpoint is taking long time
[ https://issues.apache.org/jira/browse/FLINK-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205396#comment-17205396 ] Piotr Nowojski commented on FLINK-18662: I’ve found an issue with “processed data during alignment” metric. For unaligned checkpoints the metrics that would be the most interesting are: 1. persisted in-flight data in the ckeckpoint (that’s easy to do) 2. something to compare, how persisted in-flight data compare to processed data during the time in-flight data were being persisted If number of processed data in 2. is close to 1., UC do not make much sense. But the number that we are looking for here, is not “processed during alignment”, but “processed during checkpoint” (including the async phase). "processed during alignment" would be the amount of data processed between first and last received checkpoint barrier. So I'm going to make this adjustment in my pr (instead of "during alignment" calculate the new metrics "during checkpoint") > Provide more detailed metrics why unaligned checkpoint is taking long time > -- > > Key: FLINK-18662 > URL: https://issues.apache.org/jira/browse/FLINK-18662 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Network >Affects Versions: 1.11.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski >Priority: Critical > Fix For: 1.12.0 > > Attachments: Screenshot 2020-07-21 at 11.50.02.png > > > With unaligned checkpoint there can happen situation as in the attached > screenshot. > Task reports long end to end checkpoint time (~2h50min), ~0s sync time, > ~2h50min async time, ~0s start delay. It means that task received first > checkpoint barrier from one of the channels very quickly (~0s), sync part was > quick, but we do not know why async part was taking so long. It could be > because of three things: > # long operator state IO writes > # long spilling of in-flight data > # long time to receive the final checkpoint barrier from the last lagging > channel > First and second are probably indistinguishable and the difference between > them doesn't matter much for analyzing. However the last one is quite > different. It might be independent of the IO, and we are missing this > information. > Maybe we could report it as "alignment duration" and while we are at it, we > could also report amount of spilled in-flight data for unaligned checkpoints > as "alignment buffered"? > Ideally we should report it as new metrics, but that leaves a question how to > display it in the UI, with limited space available. Maybe it could be > reported as: > ||Alignment Buffered||Alignment Duration|| > |0 B (632 MB)|0ms (2h 49m 32s)| > Where the values in the parenthesis would come from unaligned checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18818) HadoopRenameCommitterHDFSTest.testCommitOneFile[Override: false] failed with "java.io.IOException: The stream is closed"
[
https://issues.apache.org/jira/browse/FLINK-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-18818.
Fix Version/s: 1.11.3
1.12.0
Assignee: Yun Gao
Resolution: Fixed
master: 6cfdd9b8a993e8abd2a39d0123dca061adb6dbb4
release-1.11: 495defe2df4633acaafb47e86a8787540519b4a0
> HadoopRenameCommitterHDFSTest.testCommitOneFile[Override: false] failed with
> "java.io.IOException: The stream is closed"
>
>
> Key: FLINK-18818
> URL: https://issues.apache.org/jira/browse/FLINK-18818
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Yun Gao
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.11.3
>
> Attachments: image-2020-09-28-14-40-28-859.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5177=logs=5cae8624-c7eb-5c51-92d3-4d2dacedd221=420bd9ec-164e-562e-8947-0dacde3cec91
> {code}
> 2020-08-04T20:56:51.1835382Z [ERROR] testCommitOneFile[Override:
> false](org.apache.flink.formats.hadoop.bulk.committer.HadoopRenameCommitterHDFSTest)
> Time elapsed: 0.046 s <<< ERROR!
> 2020-08-04T20:56:51.1835950Z java.io.IOException: The stream is closed
> 2020-08-04T20:56:51.1836413Z at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
> 2020-08-04T20:56:51.1836867Z at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> 2020-08-04T20:56:51.1837313Z at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> 2020-08-04T20:56:51.1837712Z at
> java.io.DataOutputStream.flush(DataOutputStream.java:123)
> 2020-08-04T20:56:51.1838116Z at
> java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> 2020-08-04T20:56:51.1838527Z at
> org.apache.hadoop.hdfs.DataStreamer.closeStream(DataStreamer.java:987)
> 2020-08-04T20:56:51.1838974Z at
> org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:839)
> 2020-08-04T20:56:51.1839404Z at
> org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:834)
> 2020-08-04T20:56:51.1839775Z Suppressed: java.io.IOException: The stream is
> closed
> 2020-08-04T20:56:51.1840184Z at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
> 2020-08-04T20:56:51.1840641Z at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> 2020-08-04T20:56:51.1841087Z at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> 2020-08-04T20:56:51.1841509Z at
> java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> 2020-08-04T20:56:51.1841910Z at
> java.io.FilterOutputStream.close(FilterOutputStream.java:159)
> 2020-08-04T20:56:51.1842207Z ... 3 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] aljoscha commented on pull request #13496: [FLINK-18818][filesystem][tests] Ignores exceptions after files written successfully in HadoopRenameCommitter test
aljoscha commented on pull request #13496: URL: https://github.com/apache/flink/pull/13496#issuecomment-702011812 Thanks a lot for this fix! I merged it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] aljoscha closed pull request #13496: [FLINK-18818][filesystem][tests] Ignores exceptions after files written successfully in HadoopRenameCommitter test
aljoscha closed pull request #13496: URL: https://github.com/apache/flink/pull/13496 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19479) Allow explicitly configuring time behaviour on KeyedStream.intervalJoin()
[
https://issues.apache.org/jira/browse/FLINK-19479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205392#comment-17205392
]
Aljoscha Krettek commented on FLINK-19479:
--
Yes, it's basically the same but for joins. Should I assign you as well?
> Allow explicitly configuring time behaviour on KeyedStream.intervalJoin()
> -
>
> Key: FLINK-19479
> URL: https://issues.apache.org/jira/browse/FLINK-19479
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Priority: Major
>
> With the deprecation of
> {{StreamExecutionEnvironment.setStreamTimeCharacteristic()}} in FLINK-19319
> we need a way of explicitly configuring the time behaviour of these join
> operations. Currently, all join operations use the characteristic to
> configure themselves.
> Alternatively, we might consider removing/deprecating these join operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19462) Checkpoint statistics for unfinished task snapshots
[ https://issues.apache.org/jira/browse/FLINK-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-19462: -- Labels: usability (was: ) > Checkpoint statistics for unfinished task snapshots > --- > > Key: FLINK-19462 > URL: https://issues.apache.org/jira/browse/FLINK-19462 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Metrics >Reporter: Nico Kruber >Priority: Major > Labels: usability > > If a checkpoint times out, there are currently no stats on the > not-yet-finished tasks in the Web UI, so you have to crawl into (debug?) logs. > It would be nice to have these incomplete stats in there instead so that you > know quickly what was going on. I could think of these ways to accomplish > this: > * the checkpoint coordinator could ask the TMs for it after failing the > checkpoint or > * the TMs could send the stats when they notice that the checkpoint is > aborted > Maybe there are more options, but I think, this improvement in general would > benefit debugging checkpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19426) Streaming File Sink end-to-end test sometimes fails with "Could not assign resource ... to current execution ..."
[
https://issues.apache.org/jira/browse/FLINK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin closed FLINK-19426.
---
Resolution: Duplicate
Duplicate of FLINK-19388.
> Streaming File Sink end-to-end test sometimes fails with "Could not assign
> resource ... to current execution ..."
> -
>
> Key: FLINK-19426
> URL: https://issues.apache.org/jira/browse/FLINK-19426
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6983=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729
> {code}
> 2020-09-26T22:16:26.9856525Z
> org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException:
> Connection for partition
> 619775973ed0f282e20f9d55d13913ab#0@bc764cd8ddf7a0cff126f51c16239658_0_1 not
> reachable.
> 2020-09-26T22:16:26.9857848Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:159)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9859168Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:336)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9860449Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:308)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9861677Z at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:95)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9862861Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.requestPartitions(StreamTask.java:542)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9864018Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.readRecoveredChannelState(StreamTask.java:507)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9865284Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:498)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9866415Z at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9867500Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:492)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9868514Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:550)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9869450Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870339Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870869Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-09-26T22:16:26.9872060Z Caused by: java.io.IOException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has been lost.
> 2020-09-26T22:16:26.9873511Z at
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:85)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9874788Z at
> org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:67)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9876084Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:156)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9876567Z ... 12 more
> 2020-09-26T22:16:26.9877477Z Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has been lost.
>
[GitHub] [flink] flinkbot edited a comment on pull request #12056: [FLINK-17502] [flink-connector-rabbitmq] RMQSource refactor
flinkbot edited a comment on pull request #12056: URL: https://github.com/apache/flink/pull/12056#issuecomment-626167900 ## CI report: * c42ef1f62396948ece865a3b8628bea229d5ffb9 UNKNOWN * ad971fd47ab3ac9e813afae9713c75986646 UNKNOWN * 4d2d57b8c2f6810c03d284fdd5807b81a2b554d7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7142) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7120) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
flinkbot edited a comment on pull request #13525: URL: https://github.com/apache/flink/pull/13525#issuecomment-701444731 ## CI report: * 7309386d95647358780ec4e8a615567f720c7987 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7134) * df45481e9c3a07bcbfac6e070bc16a38fe83f3ce Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19426) Streaming File Sink end-to-end test sometimes fails with "Could not assign resource ... to current execution ..."
[
https://issues.apache.org/jira/browse/FLINK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205370#comment-17205370
]
Andrey Zagrebin commented on FLINK-19426:
-
The error has been reproduced in the CI run
[without|https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8428=results]
FLINK-19388 fix and the CI run [with the
fix|https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8429=results]
failed with maximum allowed run time exceeded w/o the reported failure.
Hence, I think this issue is a duplicate of FLINK-19388.
> Streaming File Sink end-to-end test sometimes fails with "Could not assign
> resource ... to current execution ..."
> -
>
> Key: FLINK-19426
> URL: https://issues.apache.org/jira/browse/FLINK-19426
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6983=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729
> {code}
> 2020-09-26T22:16:26.9856525Z
> org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException:
> Connection for partition
> 619775973ed0f282e20f9d55d13913ab#0@bc764cd8ddf7a0cff126f51c16239658_0_1 not
> reachable.
> 2020-09-26T22:16:26.9857848Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:159)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9859168Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:336)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9860449Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:308)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9861677Z at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:95)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9862861Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.requestPartitions(StreamTask.java:542)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9864018Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.readRecoveredChannelState(StreamTask.java:507)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9865284Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:498)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9866415Z at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9867500Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:492)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9868514Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:550)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9869450Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870339Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870869Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-09-26T22:16:26.9872060Z Caused by: java.io.IOException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has been lost.
> 2020-09-26T22:16:26.9873511Z at
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:85)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9874788Z at
> org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:67)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9876084Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:156)
>
[jira] [Comment Edited] (FLINK-19426) Streaming File Sink end-to-end test sometimes fails with "Could not assign resource ... to current execution ..."
[
https://issues.apache.org/jira/browse/FLINK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17204727#comment-17204727
]
Andrey Zagrebin edited comment on FLINK-19426 at 10/1/20, 8:30 AM:
---
I think this issue is about 'Streaming File Sink' not '-Streaming bucketing
e2e-' (at least original test failure report).
I am checking whether it is a duplicate of FLINK-19388 by running CI
[without|https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8428=results]
FLINK-19388 fix and [with
it|https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8429=results]
for 'Streaming File Sink e2e test'.
was (Author: azagrebin):
I think this issue is about 'Streaming File Sink' not '-Streaming bucketing
e2e-' (at least original test failure report).
I am checking whether it is a duplicate of FLINK-19388 by running CI
[without|https://dev.azure.com/azagrebin/azagrebin/_build/results?buildId=311=results]
FLINK-19388 fix and [with
it|https://dev.azure.com/azagrebin/azagrebin/_build/results?buildId=312=results]
for 'Streaming File Sink e2e test'.
> Streaming File Sink end-to-end test sometimes fails with "Could not assign
> resource ... to current execution ..."
> -
>
> Key: FLINK-19426
> URL: https://issues.apache.org/jira/browse/FLINK-19426
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6983=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729
> {code}
> 2020-09-26T22:16:26.9856525Z
> org.apache.flink.runtime.io.network.partition.consumer.PartitionConnectionException:
> Connection for partition
> 619775973ed0f282e20f9d55d13913ab#0@bc764cd8ddf7a0cff126f51c16239658_0_1 not
> reachable.
> 2020-09-26T22:16:26.9857848Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:159)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9859168Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.internalRequestPartitions(SingleInputGate.java:336)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9860449Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:308)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9861677Z at
> org.apache.flink.runtime.taskmanager.InputGateWithMetrics.requestPartitions(InputGateWithMetrics.java:95)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9862861Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.requestPartitions(StreamTask.java:542)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9864018Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.readRecoveredChannelState(StreamTask.java:507)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9865284Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:498)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9866415Z at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9867500Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:492)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9868514Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:550)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9869450Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870339Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-09-26T22:16:26.9870869Z at java.lang.Thread.run(Thread.java:748)
> [?:1.8.0_265]
> 2020-09-26T22:16:26.9872060Z Caused by: java.io.IOException:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Connecting to remote task manager '/10.1.0.4:38905' has failed. This might
> indicate that the remote task manager has been lost.
> 2020-09-26T22:16:26.9873511Z at
>
[GitHub] [flink] azagrebin closed pull request #13526: [FLINK-19426][NOT FOR REVIEW] Loop CI failing test with fix and AWS credentials
azagrebin closed pull request #13526: URL: https://github.com/apache/flink/pull/13526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin closed pull request #13527: [FLINK-19426][NOT FOR REVIEW] Loop CI failing test without fix and AWS credentials
azagrebin closed pull request #13527: URL: https://github.com/apache/flink/pull/13527 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
flinkbot edited a comment on pull request #13528: URL: https://github.com/apache/flink/pull/13528#issuecomment-701916570 ## CI report: * 6c34d1439fe62b9705fac808f5f15daf556604b7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7144) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
flinkbot edited a comment on pull request #13525: URL: https://github.com/apache/flink/pull/13525#issuecomment-701444731 ## CI report: * 7309386d95647358780ec4e8a615567f720c7987 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7134) * df45481e9c3a07bcbfac6e070bc16a38fe83f3ce UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19447) HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not initialized after 200000ms"
[
https://issues.apache.org/jira/browse/FLINK-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205354#comment-17205354
]
Miklos Gergely commented on FLINK-19447:
[~dian.fu] thanks for the screenshots, never thought that it was a link. Also
thank you for your review.
> HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not
> initialized after 20ms"
> ---
>
> Key: FLINK-19447
> URL: https://issues.apache.org/jira/browse/FLINK-19447
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Tests
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=60581941-0138-53c0-39fe-86d62be5f407
> {code}
> 2020-09-28T21:52:21.2146147Z
> org.apache.flink.connector.hbase2.HBaseConnectorITCase Time elapsed: 208.382
> sec <<< ERROR!
> 2020-09-28T21:52:21.2146638Z java.io.IOException: Shutting down
> 2020-09-28T21:52:21.2147004Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
> 2020-09-28T21:52:21.2147637Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:116)
> 2020-09-28T21:52:21.2148120Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1142)
> 2020-09-28T21:52:21.2148831Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
> 2020-09-28T21:52:21.2149347Z at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1061)
> 2020-09-28T21:52:21.2149896Z at
> org.apache.flink.connector.hbase2.util.HBaseTestingClusterAutoStarter.setUp(HBaseTestingClusterAutoStarter.java:122)
> 2020-09-28T21:52:21.2150721Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-09-28T21:52:21.2151136Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-09-28T21:52:21.2151609Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-09-28T21:52:21.2152039Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-09-28T21:52:21.2152462Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-09-28T21:52:21.2152941Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-09-28T21:52:21.2153489Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-09-28T21:52:21.2153962Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-09-28T21:52:21.2154406Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-09-28T21:52:21.2154828Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-09-28T21:52:21.2155381Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
> 2020-09-28T21:52:21.2155864Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
> 2020-09-28T21:52:21.2156378Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-09-28T21:52:21.2156865Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
> 2020-09-28T21:52:21.2157458Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
> 2020-09-28T21:52:21.2157993Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
> 2020-09-28T21:52:21.2158470Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> 2020-09-28T21:52:21.2158890Z Caused by: java.lang.RuntimeException: Master
> not initialized after 20ms
> 2020-09-28T21:52:21.2159350Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
> 2020-09-28T21:52:21.2159823Z at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
> 2020-09-28T21:52:21.2160270Z at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> 2020-09-28T21:52:21.2160800Z at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
> 2020-09-28T21:52:21.2161096Z ... 22 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] miklosgergely commented on a change in pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
miklosgergely commented on a change in pull request #13525:
URL: https://github.com/apache/flink/pull/13525#discussion_r498058981
##
File path:
flink-connectors/flink-connector-hbase-2.2/src/test/java/org/apache/flink/connector/hbase2/util/HBaseTestingClusterAutoStarter.java
##
@@ -119,6 +123,9 @@ private static void initialize(Configuration c) {
@BeforeClass
public static void setUp() throws Exception {
+ // HBase 2.2.3 HBaseTestingUtility works with only a certain
range of hadoop versions
+ String hadoopVersion = System.getProperty("hadoop.version");
+ Assume.assumeTrue(HADOOP_VERSION_RANGE.contains(hadoopVersion));
Review comment:
Range is using VersionUtil::compareVersions now to compare.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] miklosgergely commented on a change in pull request #13525: [FLINK-19447][hbase] Run HBase 2.2 integration test only if hadoop version is between 2.8.0 - 3.0.3
miklosgergely commented on a change in pull request #13525: URL: https://github.com/apache/flink/pull/13525#discussion_r498058649 ## File path: flink-connectors/flink-connector-hbase-2.2/pom.xml ## @@ -45,6 +45,9 @@ under the License. maven-surefire-plugin 2.19.1 + Review comment: Indeed. Removed it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol edited a comment on pull request #381: improve doc.
zentol edited a comment on pull request #381: URL: https://github.com/apache/flink-web/pull/381#issuecomment-701959588 I do not think this really enhances the documentation. If you wish to make a contribution to Flink, please read the [contribution guide](https://flink.apache.org/contributing/how-to-contribute.html). I'm sure you can find something else to work on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol closed pull request #383: Update .gitignore
zentol closed pull request #383: URL: https://github.com/apache/flink-web/pull/383 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol closed pull request #381: improve doc.
zentol closed pull request #381: URL: https://github.com/apache/flink-web/pull/381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol commented on pull request #383: Update .gitignore
zentol commented on pull request #383: URL: https://github.com/apache/flink-web/pull/383#issuecomment-701959302 Closing due to #382. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol commented on pull request #381: improve doc.
zentol commented on pull request #381: URL: https://github.com/apache/flink-web/pull/381#issuecomment-701959588 I do not think this really enhances the documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol commented on pull request #382: Update .gitignore
zentol commented on pull request #382: URL: https://github.com/apache/flink-web/pull/382#issuecomment-701959058 The [contribution guide](https://flink.apache.org/contributing/how-to-contribute.html) should give you all the information you need ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zentol closed pull request #382: Update .gitignore
zentol closed pull request #382: URL: https://github.com/apache/flink-web/pull/382 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19485) Consider runtime-mode when creating the StreamGraph
Kostas Kloudas created FLINK-19485:
--
Summary: Consider runtime-mode when creating the StreamGraph
Key: FLINK-19485
URL: https://issues.apache.org/jira/browse/FLINK-19485
Project: Flink
Issue Type: Sub-task
Components: API / DataStream
Affects Versions: 1.12.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
Although DataStream is going to be the unified API for Batch and Streaming
applications, some operations, e.g. Sinks, may need to have different runtime
implementations depending on if they are intended to run on bounded or
unbounded data. This is not necessarily only for optimisations but also for the
exposed semantics, i.e. correctness.
So far, DataStream had a 1-to-1 mapping between an API call and an operator. In
a sense, the DataStream API was an "explicit" API. With this addition, we will
decouple the API calls from the actual runtime implementations and thus allow
different operations to have more than one runtime implementations, depending
(for now) on the {{execution.runtime-mode}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19392) Detect the execution mode based on the sources in the job.
[
https://issues.apache.org/jira/browse/FLINK-19392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-19392.
--
Resolution: Implemented
Merged on master with c2e3399200010527e06e698b191bd5290f1d2ce4 and
14954490598863d2ede22eaac0282d90d206431e
> Detect the execution mode based on the sources in the job.
> --
>
> Key: FLINK-19392
> URL: https://issues.apache.org/jira/browse/FLINK-19392
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> As part of FLIP-134, we introduce the option {{execution.runtime-mode}} which
> can take the values: BATCH, STREAMING, and AUTOMATIC.
> In case of the latter, the system will scan the sources and detect if the job
> is to be execute either using batch scheduling or streaming. If all the
> sources are bounded, the system will go with BATCH, if at least one is
> unbounded, then the system will go with STREAMING.
> This issue targets introducing the logic of detecting the runtime mode based
> on the sources without exposing it yet to the user. The latter will happen in
> a follow-up issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19392) Detect the execution mode based on the sources in the job.
[
https://issues.apache.org/jira/browse/FLINK-19392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas updated FLINK-19392:
---
Fix Version/s: 1.12.0
> Detect the execution mode based on the sources in the job.
> --
>
> Key: FLINK-19392
> URL: https://issues.apache.org/jira/browse/FLINK-19392
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> As part of FLIP-134, we introduce the option {{execution.runtime-mode}} which
> can take the values: BATCH, STREAMING, and AUTOMATIC.
> In case of the latter, the system will scan the sources and detect if the job
> is to be execute either using batch scheduling or streaming. If all the
> sources are bounded, the system will go with BATCH, if at least one is
> unbounded, then the system will go with STREAMING.
> This issue targets introducing the logic of detecting the runtime mode based
> on the sources without exposing it yet to the user. The latter will happen in
> a follow-up issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u closed pull request #13502: [FLINK-19392] Add the execution.runtime-mode
kl0u closed pull request #13502: URL: https://github.com/apache/flink/pull/13502 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18988) Continuous query with LATERAL and LIMIT produces wrong result
[
https://issues.apache.org/jira/browse/FLINK-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205314#comment-17205314
]
Konstantin Knauf commented on FLINK-18988:
--
[~danny0405] Thank you for working on this issue. Are you also planning to fix
this for Flink 1.11 or only master?
> Continuous query with LATERAL and LIMIT produces wrong result
> -
>
> Key: FLINK-18988
> URL: https://issues.apache.org/jira/browse/FLINK-18988
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: Fabian Hueske
>Assignee: Danny Chen
>Priority: Critical
> Labels: pull-request-available
>
> I was trying out the example queries provided in this blog post:
> [https://materialize.io/lateral-joins-and-demand-driven-queries/] to check if
> Flink supports the same and found that the queries were translated and
> executed but produced the wrong result.
> I used the SQL Client and Kafka (running at kafka:9092) to store the table
> data. I executed the following statements:
> {code:java}
> -- create cities table
> CREATE TABLE cities (
> name STRING NOT NULL,
> state STRING NOT NULL,
> pop INT NOT NULL
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'cities',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'mygroup',
> 'scan.startup.mode' = 'earliest-offset',
> 'format' = 'json'
> );
> -- fill cities table
> INSERT INTO cities VALUES
> ('Los_Angeles', 'CA', 3979576),
> ('Phoenix', 'AZ', 1680992),
> ('Houston', 'TX', 2320268),
> ('San_Diego', 'CA', 1423851),
> ('San_Francisco', 'CA', 881549),
> ('New_York', 'NY', 8336817),
> ('Dallas', 'TX', 1343573),
> ('San_Antonio', 'TX', 1547253),
> ('San_Jose', 'CA', 1021795),
> ('Chicago', 'IL', 2695598),
> ('Austin', 'TX', 978908);
> -- execute query
> SELECT state, name
> FROM
> (SELECT DISTINCT state FROM cities) states,
> LATERAL (
> SELECT name, pop
> FROM cities
> WHERE state = states.state
> ORDER BY pop
> DESC LIMIT 3
> );
> -- result
> state name
>CA Los_Angeles
>NY New_York
>IL Chicago
> -- expected result
> state | name
> --+-
> TX | Dallas
> AZ | Phoenix
> IL | Chicago
> TX | Houston
> CA | San_Jose
> NY | New_York
> CA | San_Diego
> CA | Los_Angeles
> TX | San_Antonio
> {code}
> As you can see from the query result, Flink computes the top3 cities over all
> states, not for every state individually. Hence, I assume that this is a bug
> in the query optimizer or one of the rewriting rules.
> There are two valid ways to solve this issue:
> * Fixing the rewriting rules / optimizer (obviously preferred)
> * Disabling this feature and throwing an exception
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * 4824a2e5258649246c5372200dd44fb61e20d012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7143) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
flinkbot edited a comment on pull request #13528: URL: https://github.com/apache/flink/pull/13528#issuecomment-701916570 ## CI report: * 6c34d1439fe62b9705fac808f5f15daf556604b7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7144) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19400) Removed unused BufferPoolOwner
[
https://issues.apache.org/jira/browse/FLINK-19400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205297#comment-17205297
]
Arvid Heise commented on FLINK-19400:
-
Merged into master as part of FLINK-16972.
> Removed unused BufferPoolOwner
> --
>
> Key: FLINK-19400
> URL: https://issues.apache.org/jira/browse/FLINK-19400
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Arvid Heise
>Assignee: Arvid Heise
>Priority: Major
>
> {{BufferPoolOwner}} does not have any production usages and just complicates
> a few tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19400) Removed unused BufferPoolOwner
[
https://issues.apache.org/jira/browse/FLINK-19400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise closed FLINK-19400.
---
Resolution: Fixed
> Removed unused BufferPoolOwner
> --
>
> Key: FLINK-19400
> URL: https://issues.apache.org/jira/browse/FLINK-19400
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Arvid Heise
>Assignee: Arvid Heise
>Priority: Major
>
> {{BufferPoolOwner}} does not have any production usages and just complicates
> a few tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16972) Fix non-blocking output availability logic.
[ https://issues.apache.org/jira/browse/FLINK-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-16972: Affects Version/s: 1.12.0 > Fix non-blocking output availability logic. > --- > > Key: FLINK-16972 > URL: https://issues.apache.org/jira/browse/FLINK-16972 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.11.0, 1.12.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Non-blocking output is currently blocking, as unavailability in > `LocalBufferPool` is only signaled when the first unavailable buffer is being > requested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16972) Fix non-blocking output availability logic.
[ https://issues.apache.org/jira/browse/FLINK-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205295#comment-17205295 ] Arvid Heise commented on FLINK-16972: - Merged into master as e17dbab24f4f71c5472d27267e938791686e45c3. > Fix non-blocking output availability logic. > --- > > Key: FLINK-16972 > URL: https://issues.apache.org/jira/browse/FLINK-16972 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.11.0 >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Non-blocking output is currently blocking, as unavailability in > `LocalBufferPool` is only signaled when the first unavailable buffer is being > requested. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #13499: [FLINK-16972][network] LocalBufferPool eagerly fetches global segments to ensure proper availability.
AHeise merged pull request #13499: URL: https://github.com/apache/flink/pull/13499 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] shudhanshubisht08 opened a new pull request #383: Update .gitignore
shudhanshubisht08 opened a new pull request #383: URL: https://github.com/apache/flink-web/pull/383 Nice work. Can I Do some Changes in this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] shudhanshubisht08 opened a new pull request #382: Update .gitignore
shudhanshubisht08 opened a new pull request #382: URL: https://github.com/apache/flink-web/pull/382 I want to make some contribution to this project This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
flinkbot commented on pull request #13528: URL: https://github.com/apache/flink/pull/13528#issuecomment-701916570 ## CI report: * 6c34d1439fe62b9705fac808f5f15daf556604b7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13521: [FLINK-19472] Implement a one input sorting DataInput
flinkbot edited a comment on pull request #13521: URL: https://github.com/apache/flink/pull/13521#issuecomment-701282459 ## CI report: * b127687d16b67d98077b0fb108a024e6d7b2dca3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7135) * 4824a2e5258649246c5372200dd44fb61e20d012 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-19249:
---
Issue Type: Improvement (was: Bug)
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-19249:
---
Priority: Critical (was: Blocker)
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Critical
> Fix For: 1.12.0, 1.11.3
>
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken
[
https://issues.apache.org/jira/browse/FLINK-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-19249:
---
Fix Version/s: (was: 1.11.3)
> Job would wait sometime(~10 min) before failover if some connection broken
> --
>
> Key: FLINK-19249
> URL: https://issues.apache.org/jira/browse/FLINK-19249
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: Congxian Qiu(klion26)
>Priority: Critical
> Fix For: 1.12.0
>
>
> {quote}encountered this error on 1.7, after going through the master code, I
> think the problem is still there
> {quote}
> When the network environment is not so good, the connection between the
> server and the client may be disconnected innocently. After the
> disconnection, the server will receive the IOException such as below
> {code:java}
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> then release the view reader.
> But the job would not fail until the downstream detect the disconnection
> because of {{channelInactive}} later(~10 min). between such time, the job can
> still process data, but the broken channel can't transfer any data or event,
> so snapshot would fail during this time. this will cause the job to replay
> many data after failover.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
flinkbot commented on pull request #13528: URL: https://github.com/apache/flink/pull/13528#issuecomment-701909456 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6c34d1439fe62b9705fac808f5f15daf556604b7 (Thu Oct 01 06:08:41 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19477) Translate page 'python_table_api_connectors' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19477: --- Labels: pull-request-available (was: ) > Translate page 'python_table_api_connectors' into Chinese > - > > Key: FLINK-19477 > URL: https://issues.apache.org/jira/browse/FLINK-19477 > Project: Flink > Issue Type: Improvement > Components: API / Python, chinese-translation >Affects Versions: 1.11.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > The page url > [python_table_api_connectors|https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/python/user-guide/table/python_table_api_connectors.html] > The doc is located in > /dev/python/user-guide/table/python_table_api_connectors.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangxlong opened a new pull request #13528: [FLINK-19477][docs-zh][python] Translate page "python_table_api_connectors" into Chinese
wangxlong opened a new pull request #13528: URL: https://github.com/apache/flink/pull/13528 ## What is the purpose of the change Translate page "python_table_api_connectors" into Chinese ## Brief change log Translate page "python_table_api_connectors" into Chinese ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
