[GitHub] [flink] dianfu commented on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
dianfu commented on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-711121589 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * ae3f85a1f5601225f20bd08cef1b176f692e9e84 UNKNOWN * 9e9eddc7dbfb72a452ff98618e9fa7e5d624f5da Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7795) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19692) Can't restore feedback channel from savepoint
Antti Kaikkonen created FLINK-19692:
---
Summary: Can't restore feedback channel from savepoint
Key: FLINK-19692
URL: https://issues.apache.org/jira/browse/FLINK-19692
Project: Flink
Issue Type: Bug
Components: API / DataStream, API / State Processor, Stateful
Functions
Affects Versions: 1.11.2
Reporter: Antti Kaikkonen
When using the new statefun-flink-datastream integration the following error is
thrown by the *feedback -> union* task when trying to restore from a savepoint:
{code:java}
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:204)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.IOException: position out of bounds
at
org.apache.flink.runtime.state.StatePartitionStreamProvider.getStream(StatePartitionStreamProvider.java:58)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.internalTimeServiceManager(StreamTaskStateInitializerImpl.java:235)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:167)
... 9 more
Caused by: java.io.IOException: position out of bounds
at
org.apache.flink.runtime.state.memory.ByteStreamStateHandle$ByteStateHandleInputStream.seek(ByteStreamStateHandle.java:124)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl$KeyGroupStreamIterator.next(StreamTaskStateInitializerImpl.java:442)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl$KeyGroupStreamIterator.next(StreamTaskStateInitializerImpl.java:395)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.internalTimeServiceManager(StreamTaskStateInitializerImpl.java:228)
... 10 more
{code}
The error is only thrown when the feedback channel has been used.
I have tested with the [example
application|https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-flink-datastream-example/src/main/java/org/apache/flink/statefun/examples/datastream/Example.java]
and the error is thrown only if it is modified to actually use the feedback
channel. I simply modified the invoke method to sometimes forward the greeting
to a random name:
{code:java}
@Override
public void invoke(Context context, Object input) {
int seen = seenCount.updateAndGet(MyFunction::increment);
context.send(GREETINGS, String.format("Hello %s at the %d-th time", input,
seen));
String[] names = {"Stephan", "Igal", "Gordon", "Seth", "Marta"};
ThreadLocalRandom random = ThreadLocalRandom.current();
int index = random.nextInt(names.length);
final String name2 = names[index];
if (random.nextDouble() < 0.5) context.send(new Address(GREET, name2), input);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13300: [FLINK-19077][table-runtime] Import process time temporal join operator.
flinkbot edited a comment on pull request #13300: URL: https://github.com/apache/flink/pull/13300#issuecomment-684957567 ## CI report: * b32accb0c729ab361a8becf74808427f103bdeb0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7797) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13300: [FLINK-19077][table-runtime] Import process time temporal join operator.
flinkbot edited a comment on pull request #13300: URL: https://github.com/apache/flink/pull/13300#issuecomment-684957567 ## CI report: * 47e4cd9eca1c2772d67e484cfe3f732b4008b188 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6104) * b32accb0c729ab361a8becf74808427f103bdeb0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7797) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
flinkbot edited a comment on pull request #13672: URL: https://github.com/apache/flink/pull/13672#issuecomment-711107288 ## CI report: * 3f25a527ec7c3238365f0746323c02f0f35252cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7794) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13300: [FLINK-19077][table-runtime] Import process time temporal join operator.
flinkbot edited a comment on pull request #13300: URL: https://github.com/apache/flink/pull/13300#issuecomment-684957567 ## CI report: * 47e4cd9eca1c2772d67e484cfe3f732b4008b188 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6104) * b32accb0c729ab361a8becf74808427f103bdeb0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13673: [FLINK-19689][yarn][test] Fix TaskExecutorProcessSpecContainerResourcePriorityAdapterTest for Hadoop 2.10+.
flinkbot edited a comment on pull request #13673: URL: https://github.com/apache/flink/pull/13673#issuecomment-72023 ## CI report: * b5e00b966be2e1295430a64ccbaf75ce5547c6d5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7796) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13673: [FLINK-19689][yarn][test] Fix TaskExecutorProcessSpecContainerResourcePriorityAdapterTest for Hadoop 2.10+.
flinkbot commented on pull request #13673: URL: https://github.com/apache/flink/pull/13673#issuecomment-72023 ## CI report: * b5e00b966be2e1295430a64ccbaf75ce5547c6d5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * ae3f85a1f5601225f20bd08cef1b176f692e9e84 UNKNOWN * ea5aa5217447669b700f1c645a41db57e1e01350 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7761) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7737) * 9e9eddc7dbfb72a452ff98618e9fa7e5d624f5da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7795) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13673: [FLINK-19689][yarn][test] Fix TaskExecutorProcessSpecContainerResourcePriorityAdapterTest for Hadoop 2.10+.
flinkbot commented on pull request #13673: URL: https://github.com/apache/flink/pull/13673#issuecomment-71630 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b5e00b966be2e1295430a64ccbaf75ce5547c6d5 (Sun Oct 18 03:34:25 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13673: [FLINK-19689][yarn][test] Fix TaskExecutorProcessSpecContainerResourcePriorityAdapterTest for Hadoop 2.10+.
xintongsong commented on pull request #13673: URL: https://github.com/apache/flink/pull/13673#issuecomment-71491 @KarmaGYZ Could you help take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19689:
---
Labels: pull-request-available test-stability (was: test-stability)
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Xintong Song
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
>
[GitHub] [flink] xintongsong opened a new pull request #13673: [FLINK-19689][yarn][test] Fix TaskExecutorProcessSpecContainerResourcePriorityAdapterTest for Hadoop 2.10+.
xintongsong opened a new pull request #13673: URL: https://github.com/apache/flink/pull/13673 ## What is the purpose of the change This PR fixes the test failure of `TaskExecutorProcessSpecContainerResourcePriorityAdapterTest` when running with Hadoop 2.10+. ## Verifying this change Verified with Hadoop 2.4.1, 2.10.0, 3.1.3 and 3.3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216085#comment-17216085
]
Xintong Song commented on FLINK-19689:
--
The problem is that we try to set a {{testing-external-resource}} in the test
case, while Hadoop 2.10+ checks it against the {{resource-types.xml}} and finds
it not defined.
The solution is to add {{flink-yarn/src/test/resoruces/resource-types.xml}}
with the testing resource type included.
I'm opening a PR to fix this.
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13492: [FLINK-19181][python] Make python processes respect the calculated managed memory fraction
flinkbot edited a comment on pull request #13492: URL: https://github.com/apache/flink/pull/13492#issuecomment-699626263 ## CI report: * af73558800083e341dcd9c89819cd00b0572696b UNKNOWN * 59af0e9ec80df5072e44576882f8659db631e424 UNKNOWN * ae3f85a1f5601225f20bd08cef1b176f692e9e84 UNKNOWN * ea5aa5217447669b700f1c645a41db57e1e01350 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7761) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7737) * 9e9eddc7dbfb72a452ff98618e9fa7e5d624f5da UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19691) Expose `CONNECTION_CHECK_TIMEOUT_SECONDS` as a configurable option in Jdbc connector
hailong wang created FLINK-19691: Summary: Expose `CONNECTION_CHECK_TIMEOUT_SECONDS` as a configurable option in Jdbc connector Key: FLINK-19691 URL: https://issues.apache.org/jira/browse/FLINK-19691 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Affects Versions: 1.11.0 Reporter: hailong wang Fix For: 1.12.0 Jdbc connector can check whether the connection is valid, and the timeout is 60 second [FLINK-16681]. But the fixed timeout of 60 second sometimes too long to wait. In some scenes, I just want to wait for 5 second and fast failed and restarting if it is still invalid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
flinkbot edited a comment on pull request #13672: URL: https://github.com/apache/flink/pull/13672#issuecomment-711107288 ## CI report: * 3f25a527ec7c3238365f0746323c02f0f35252cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7794) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
flinkbot commented on pull request #13672: URL: https://github.com/apache/flink/pull/13672#issuecomment-711107288 ## CI report: * 3f25a527ec7c3238365f0746323c02f0f35252cb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216080#comment-17216080
]
Xintong Song commented on FLINK-19689:
--
Thanks for the pointer, [~dian.fu]. I'm looking into this.
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
>
[GitHub] [flink] flinkbot commented on pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
flinkbot commented on pull request #13672: URL: https://github.com/apache/flink/pull/13672#issuecomment-711106378 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3f25a527ec7c3238365f0746323c02f0f35252cb (Sun Oct 18 02:13:21 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong opened a new pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
wangxlong opened a new pull request #13672: URL: https://github.com/apache/flink/pull/13672 ## What is the purpose of the change Translate page "table_environment.zh.md" into Chinese ## Brief change log Translate page "table_environment.zh.md" into Chinese ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no / don't know) - The runtime per-record code paths (performance sensitive): (no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no / don't know) - The S3 file system connector: (no / don't know) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19660) Translate page 'table_environment.zh.md' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19660: --- Labels: pull-request-available (was: ) > Translate page 'table_environment.zh.md' into Chinese > - > > Key: FLINK-19660 > URL: https://issues.apache.org/jira/browse/FLINK-19660 > Project: Flink > Issue Type: Improvement > Components: API / Python, chinese-translation, Documentation >Affects Versions: 1.12.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > The page url is > [TableEnvironment|https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/python/table-api-users-guide/table_environment.html] > The url is located in > /dev/python/table-api-users-guide/table_environment.zh.md > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reassigned FLINK-19689:
Assignee: Xintong Song
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>
[jira] [Commented] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216075#comment-17216075
]
Dian Fu commented on FLINK-19689:
-
cc [~xintongsong]
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-10-16T21:48:16.5374072Z at
>
[jira] [Updated] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19689:
Priority: Critical (was: Major)
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-10-16T21:48:16.5374072Z at
>
[jira] [Commented] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216074#comment-17216074
]
Dian Fu commented on FLINK-19689:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7791=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z
[jira] [Commented] (FLINK-17424) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to download error
[
https://issues.apache.org/jira/browse/FLINK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216072#comment-17216072
]
Dian Fu commented on FLINK-17424:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7791=logs=08866332-78f7-59e4-4f7e-49a56faa3179=7f606211-1454-543c-70ab-c7a028a1ce8c
> SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to
> download error
>
>
> Key: FLINK-17424
> URL: https://issues.apache.org/jira/browse/FLINK-17424
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yu Li
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> `SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1)` failed in
> release-1.10 crone job with below error:
> {noformat}
> Preparing Elasticsearch(version=7)...
> Downloading Elasticsearch from
> https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-linux-x86_64.tar.gz
> ...
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 4 276M4 13.3M0 0 28.8M 0 0:00:09 --:--:-- 0:00:09 28.8M
> 42 276M 42 117M0 0 80.7M 0 0:00:03 0:00:01 0:00:02 80.7M
> 70 276M 70 196M0 0 79.9M 0 0:00:03 0:00:02 0:00:01 79.9M
> 89 276M 89 248M0 0 82.3M 0 0:00:03 0:00:03 --:--:-- 82.4M
> curl: (56) GnuTLS recv error (-54): Error in the pull function.
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
> 0curl: (7) Failed to connect to localhost port 9200: Connection refused
> [FAIL] Test script contains errors.
> {noformat}
> https://api.travis-ci.org/v3/job/680222168/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19690) ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect failed with "expected:<1> but was:<2>"
[
https://issues.apache.org/jira/browse/FLINK-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19690:
Labels: test-stability (was: )
> ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect failed with
> "expected:<1> but was:<2>"
> -
>
> Key: FLINK-19690
> URL: https://issues.apache.org/jira/browse/FLINK-19690
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.11.2
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7792=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=603cb7fd-6f38-5c99-efca-877e1439232f
> {code}
> [ERROR]
> testWriteParquetAvroReflect(org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase)
> Time elapsed: 1.407 s <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase.validateResults(ParquetStreamingFileSinkITCase.java:161)
>
> at
> org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect(ParquetStreamingFileSinkITCase.java:153)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19690) ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect failed with "expected:<1> but was:<2>"
Dian Fu created FLINK-19690:
---
Summary:
ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect failed with
"expected:<1> but was:<2>"
Key: FLINK-19690
URL: https://issues.apache.org/jira/browse/FLINK-19690
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.11.2
Reporter: Dian Fu
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7792=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=603cb7fd-6f38-5c99-efca-877e1439232f
{code}
[ERROR]
testWriteParquetAvroReflect(org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase)
Time elapsed: 1.407 s <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase.validateResults(ParquetStreamingFileSinkITCase.java:161)
at
org.apache.flink.formats.parquet.avro.ParquetStreamingFileSinkITCase.testWriteParquetAvroReflect(ParquetStreamingFileSinkITCase.java:153)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19469) HBase connector 2.2 failed to download dependencies "org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT"
[
https://issues.apache.org/jira/browse/FLINK-19469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-19469.
---
Fix Version/s: 1.12.0
Resolution: Fixed
Fixed in master(1.12.0) via ce143f26b8236ddfca77c270fab31c98e906356a
> HBase connector 2.2 failed to download dependencies
> "org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT"
>
>
> Key: FLINK-19469
> URL: https://issues.apache.org/jira/browse/FLINK-19469
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7093=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> {code}
> 2020-09-29T20:59:24.8085970Z [ERROR] Failed to execute goal on project
> flink-connector-hbase-2.2_2.11: Could not resolve dependencies for project
> org.apache.flink:flink-connector-hbase-2.2_2.11:jar:1.12-SNAPSHOT: Failed to
> collect dependencies at org.apache.hbase:hbase-server:jar:tests:2.2.3 ->
> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 ->
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact
> descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not
> transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to
> jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 503 , ReasonPhrase:Service Unavailable: Back-end server is
> at capacity. -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dianfu merged pull request #13622: [FLINK-19469][hbase] Exclude org.glassfish:javax.el transitive dependency from hbase-2.2 connector as it is unreliable
dianfu merged pull request #13622: URL: https://github.com/apache/flink/pull/13622 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] miklosgergely commented on pull request #13622: [FLINK-19469][hbase] Exclude org.glassfish:javax.el transitive dependency from hbase-2.2 connector as it is unreliable
miklosgergely commented on pull request #13622: URL: https://github.com/apache/flink/pull/13622#issuecomment-711088728 @dianfu could you please merge this one? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13555: [FLINK-17613][e2e][k8s] Test against multiple k8s versions
flinkbot edited a comment on pull request #13555: URL: https://github.com/apache/flink/pull/13555#issuecomment-704987370 ## CI report: * dbb8f2d60f4abd5db43d72d34352fdb2083af370 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7646) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7675) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7726) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7789) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7769) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13622: [FLINK-19469][hbase] Exclude org.glassfish:javax.el transitive dependency from hbase-2.2 connector as it is unreliable
flinkbot edited a comment on pull request #13622: URL: https://github.com/apache/flink/pull/13622#issuecomment-708066181 ## CI report: * 4fa1aef70494d800be2d9cd5c9345ecc1700b692 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7790) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #13598: [FLINK-18915][filesystem][orc] Fix the incorrect memory management behavior by using uuid fake path for orc writer
kl0u commented on pull request #13598: URL: https://github.com/apache/flink/pull/13598#issuecomment-711060518 No problem @gaoyunhaii ! Thanks for the work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #13598: [FLINK-18915][filesystem][orc] Fix the incorrect memory management behavior by using uuid fake path for orc writer
gaoyunhaii commented on pull request #13598: URL: https://github.com/apache/flink/pull/13598#issuecomment-711048258 Very thanks @kl0u for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13622: [FLINK-19469][hbase] Exclude org.glassfish:javax.el transitive dependency from hbase-2.2 connector as it is unreliable
flinkbot edited a comment on pull request #13622: URL: https://github.com/apache/flink/pull/13622#issuecomment-708066181 ## CI report: * 428b6634795fd9eae35911cf468b96b2542b7e75 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7780) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7778) * 4fa1aef70494d800be2d9cd5c9345ecc1700b692 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7790) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13622: [FLINK-19469][hbase] Exclude org.glassfish:javax.el transitive dependency from hbase-2.2 connector as it is unreliable
flinkbot edited a comment on pull request #13622: URL: https://github.com/apache/flink/pull/13622#issuecomment-708066181 ## CI report: * 428b6634795fd9eae35911cf468b96b2542b7e75 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7780) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7778) * 4fa1aef70494d800be2d9cd5c9345ecc1700b692 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13555: [FLINK-17613][e2e][k8s] Test against multiple k8s versions
flinkbot edited a comment on pull request #13555: URL: https://github.com/apache/flink/pull/13555#issuecomment-704987370 ## CI report: * dbb8f2d60f4abd5db43d72d34352fdb2083af370 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7646) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7675) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7726) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7789) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7769) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13555: [FLINK-17613][e2e][k8s] Test against multiple k8s versions
flinkbot edited a comment on pull request #13555: URL: https://github.com/apache/flink/pull/13555#issuecomment-704987370 ## CI report: * dbb8f2d60f4abd5db43d72d34352fdb2083af370 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7646) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7675) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7769) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7726) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7789) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] scottxusayhi commented on pull request #13555: [FLINK-17613][e2e][k8s] Test against multiple k8s versions
scottxusayhi commented on pull request #13555: URL: https://github.com/apache/flink/pull/13555#issuecomment-711025258 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] scottxusayhi removed a comment on pull request #13555: [FLINK-17613][e2e][k8s] Test against multiple k8s versions
scottxusayhi removed a comment on pull request #13555: URL: https://github.com/apache/flink/pull/13555#issuecomment-710100564 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong removed a comment on pull request #13643: [Hotfix][CheckStyle][Min] Fix DummyInputFormat class lacks Javadoc
wangxlong removed a comment on pull request #13643: URL: https://github.com/apache/flink/pull/13643#issuecomment-708999445 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13667: [FLINK-19679] Deduplicate code between CheckpointBarrierUnaligner and CheckpointBarrierAligner
flinkbot edited a comment on pull request #13667: URL: https://github.com/apache/flink/pull/13667#issuecomment-710095324 ## CI report: * de848b220414a40d24a3b6ec2656dcbe1ad30f3a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7788) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13671: [FLINK-19625][table-planner] Introduce multi-input exec node
flinkbot edited a comment on pull request #13671: URL: https://github.com/apache/flink/pull/13671#issuecomment-710779251 ## CI report: * 745d079d12088ca1faeb89a7b7974c8bbea383b5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7787) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13643: [Hotfix][CheckStyle][Min] Fix DummyInputFormat class lacks Javadoc
flinkbot edited a comment on pull request #13643: URL: https://github.com/apache/flink/pull/13643#issuecomment-708870329 ## CI report: * 5fe6724b54691b8283fb6ddce689d53a8a420fda Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7786) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13602: [FLINK-19346][coordination] Generate and put ClusterPartitionDescriptor of ClusterPartition in JobResult when job finishes
flinkbot edited a comment on pull request #13602: URL: https://github.com/apache/flink/pull/13602#issuecomment-70743 ## CI report: * 5bd17fad8f85ce110218ef2ec98912b138db03e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7784) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7763) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] carp84 commented on pull request #131: [FLINK-18968] Translate README.md to Chinese
carp84 commented on pull request #131: URL: https://github.com/apache/flink-statefun/pull/131#issuecomment-710845269 Thanks for the great job @klion26 ! @billyrrr could you update the PR according to the review suggestions and [translation specification](https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications)? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13670: [hotfix][docs]Fix typos in docs
flinkbot edited a comment on pull request #13670: URL: https://github.com/apache/flink/pull/13670#issuecomment-710731909 ## CI report: * 66c0f3542d0f0898a2eba8013ab4a390f35c263d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7785) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #13631: [FLINK-19639][table sql/planner]Support SupportsNestedProjectionPushD…
godfreyhe commented on a change in pull request #13631:
URL: https://github.com/apache/flink/pull/13631#discussion_r506915187
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/RexNodeExtractor.scala
##
@@ -256,7 +256,15 @@ class RefFieldAccessorVisitor(usedFields: Array[Int])
extends RexVisitorImpl[Uni
if (right.length < left.length) {
false
} else {
- right.take(left.length).equals(left)
+ right.take(left.length).zip(left).foldLeft(true) {
+(ans, fields) => {
+ if (ans) {
+fields._1.equals(fields._2)
Review comment:
It's better we does not use `_1`, `_2`, which makes the code hard to
read. we can use use `case` match with meaningful name, just like:
```
right.take(left.length).zip(left).foldLeft(true) {
case (ans, (rName, lName)) => {
if (ans) {
lName.equals(rName)
} else {
false
}
}
}
```
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TableSchemaUtils.java
##
@@ -75,10 +79,46 @@ public static TableSchema projectSchema(TableSchema
tableSchema, int[][] project
checkArgument(containsPhysicalColumnsOnly(tableSchema),
"Projection is only supported for physical columns.");
TableSchema.Builder schemaBuilder = TableSchema.builder();
List tableColumns = tableSchema.getTableColumns();
+ Map nameDomain = new HashMap<>();
+ String exceptionTemplate = "Get name conflicts for origin
fields %s and %s with new name `%s`. " +
+ "When pushing projection into scan, we will
concatenate top level names with delimiter '_'. " +
+ "Please rename the origin field names when
creating table.";
+ String originFullyQualifiedName;
+ String newName;
for (int[] fieldPath : projectedFields) {
- checkArgument(fieldPath.length == 1, "Nested projection
push down is not supported yet.");
- TableColumn column = tableColumns.get(fieldPath[0]);
- schemaBuilder.field(column.getName(), column.getType());
+ if (fieldPath.length == 1) {
+ TableColumn column =
tableColumns.get(fieldPath[0]);
+ newName = column.getName();
+ originFullyQualifiedName =
String.format("`%s`", column.getName());
+ if (nameDomain.containsKey(column.getName())) {
+ throw new TableException(
Review comment:
how about resolve the conflicts through adding postfix ?
##
File path:
flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRuleTest.java
##
@@ -70,41 +69,40 @@ public void setup() {
" 'bounded' = 'true'\n" +
")";
util().tableEnv().executeSql(ddl2);
- }
-
- @Override
- public void testNestedProject() {
- expectedException().expect(TableException.class);
- expectedException().expectMessage("Nested projection push down
is unsupported now.");
- testNestedProject(true);
- }
- @Test
- public void testNestedProjectDisabled() {
- testNestedProject(false);
- }
-
- private void testNestedProject(boolean nestedProjectionSupported) {
- String ddl =
+ String ddl3 =
"CREATE TABLE NestedTable (\n" +
" id int,\n" +
- " deepNested row, nested2 row>,\n" +
- " nested row,\n" +
+ " deepNested row, `nested2.` row>,\n" +
+ " nested row,\n" +
" name string\n" +
") WITH (\n" +
" 'connector' = 'values',\n" +
- " 'nested-projection-supported'
= '" + nestedProjectionSupported + "',\n" +
+ " 'nested-projection-supported'
= 'true'," +
" 'bounded' = 'true'\n" +
")";
- util().tableEnv().executeSql(ddl);
+ util().tableEnv().executeSql(ddl3);
+ }
+ @Override
+ @Test
+ public
[jira] [Commented] (FLINK-19469) HBase connector 2.2 failed to download dependencies "org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT"
[
https://issues.apache.org/jira/browse/FLINK-19469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215833#comment-17215833
]
Robert Metzger commented on FLINK-19469:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7773=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
> HBase connector 2.2 failed to download dependencies
> "org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT"
>
>
> Key: FLINK-19469
> URL: https://issues.apache.org/jira/browse/FLINK-19469
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7093=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> {code}
> 2020-09-29T20:59:24.8085970Z [ERROR] Failed to execute goal on project
> flink-connector-hbase-2.2_2.11: Could not resolve dependencies for project
> org.apache.flink:flink-connector-hbase-2.2_2.11:jar:1.12-SNAPSHOT: Failed to
> collect dependencies at org.apache.hbase:hbase-server:jar:tests:2.2.3 ->
> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 ->
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact
> descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not
> transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to
> jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 503 , ReasonPhrase:Service Unavailable: Back-end server is
> at capacity. -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
Robert Metzger created FLINK-19689:
--
Summary:
TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
Key: FLINK-19689
URL: https://issues.apache.org/jira/browse/FLINK-19689
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.12.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
{code}
2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
2020-10-16T21:48:16.5355440Z [ERROR]
testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
Time elapsed: 0.005 s <<< ERROR!
2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External resource
testing-external-resource is not supported by the Yarn cluster.
2020-10-16T21:48:16.5356834Zat
org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
2020-10-16T21:48:16.5357490Zat
org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
2020-10-16T21:48:16.5358469Zat
org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
2020-10-16T21:48:16.5359460Zat
org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
2020-10-16T21:48:16.5360791Zat
org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
2020-10-16T21:48:16.5361437Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-10-16T21:48:16.5362144Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-10-16T21:48:16.5363032Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-10-16T21:48:16.5363704Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-10-16T21:48:16.5364339Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-10-16T21:48:16.5365108Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-10-16T21:48:16.5365880Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-10-16T21:48:16.5366598Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-10-16T21:48:16.5367162Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-10-16T21:48:16.5367560Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-10-16T21:48:16.5367943Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-10-16T21:48:16.5368543Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-10-16T21:48:16.5369034Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-10-16T21:48:16.5369455Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-10-16T21:48:16.5369859Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-10-16T21:48:16.5370276Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-10-16T21:48:16.5370676Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-10-16T21:48:16.5371091Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-10-16T21:48:16.5371497Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-10-16T21:48:16.5371921Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-10-16T21:48:16.5372422Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2020-10-16T21:48:16.5373073Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-10-16T21:48:16.5373554Zat
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2020-10-16T21:48:16.5374072Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-10-16T21:48:16.5374660Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-10-16T21:48:16.5375128Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-10-16T21:48:16.5375585Zat
[jira] [Updated] (FLINK-19689) TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
[
https://issues.apache.org/jira/browse/FLINK-19689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-19689:
---
Fix Version/s: 1.12.0
> TaskExecutorProcessSpecContainerResourcePriorityAdapterTest fails
> -
>
> Key: FLINK-19689
> URL: https://issues.apache.org/jira/browse/FLINK-19689
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354
> {code}
> 2020-10-16T21:48:16.5354451Z [ERROR] Tests run: 8, Failures: 0, Errors: 1,
> Skipped: 1, Time elapsed: 0.022 s <<< FAILURE! - in
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest
> 2020-10-16T21:48:16.5355440Z [ERROR]
> testExternalResource(org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest)
> Time elapsed: 0.005 s <<< ERROR!
> 2020-10-16T21:48:16.5356271Z java.lang.IllegalStateException: External
> resource testing-external-resource is not supported by the Yarn cluster.
> 2020-10-16T21:48:16.5356834Z at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:220)
> 2020-10-16T21:48:16.5357490Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.validateExternalResourceConfigs(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:85)
> 2020-10-16T21:48:16.5358469Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapter.(TaskExecutorProcessSpecContainerResourcePriorityAdapter.java:57)
> 2020-10-16T21:48:16.5359460Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.getAdapterWithExternalResources(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:170)
> 2020-10-16T21:48:16.5360791Z at
> org.apache.flink.yarn.TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.testExternalResource(TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java:129)
> 2020-10-16T21:48:16.5361437Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-10-16T21:48:16.5362144Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-10-16T21:48:16.5363032Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-10-16T21:48:16.5363704Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-10-16T21:48:16.5364339Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-10-16T21:48:16.5365108Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-10-16T21:48:16.5365880Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-10-16T21:48:16.5366598Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-10-16T21:48:16.5367162Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-10-16T21:48:16.5367560Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-10-16T21:48:16.5367943Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-10-16T21:48:16.5368543Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-10-16T21:48:16.5369034Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-10-16T21:48:16.5369455Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-10-16T21:48:16.5369859Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-10-16T21:48:16.5370276Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-10-16T21:48:16.5370676Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-10-16T21:48:16.5371091Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-10-16T21:48:16.5371497Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-10-16T21:48:16.5371921Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-10-16T21:48:16.5372422Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-10-16T21:48:16.5373073Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-10-16T21:48:16.5373554Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-10-16T21:48:16.5374072Z at
>
[jira] [Commented] (FLINK-17424) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to download error
[
https://issues.apache.org/jira/browse/FLINK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215831#comment-17215831
]
Robert Metzger commented on FLINK-17424:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7776=logs=68a897ab-3047-5660-245a-cce8f83859f6=16ca2cca-2f63-5cce-12d2-d519b930a729
> SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) failed due to
> download error
>
>
> Key: FLINK-17424
> URL: https://issues.apache.org/jira/browse/FLINK-17424
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yu Li
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> `SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1)` failed in
> release-1.10 crone job with below error:
> {noformat}
> Preparing Elasticsearch(version=7)...
> Downloading Elasticsearch from
> https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-linux-x86_64.tar.gz
> ...
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 4 276M4 13.3M0 0 28.8M 0 0:00:09 --:--:-- 0:00:09 28.8M
> 42 276M 42 117M0 0 80.7M 0 0:00:03 0:00:01 0:00:02 80.7M
> 70 276M 70 196M0 0 79.9M 0 0:00:03 0:00:02 0:00:01 79.9M
> 89 276M 89 248M0 0 82.3M 0 0:00:03 0:00:03 --:--:-- 82.4M
> curl: (56) GnuTLS recv error (-54): Error in the pull function.
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
> 0curl: (7) Failed to connect to localhost port 9200: Connection refused
> [FAIL] Test script contains errors.
> {noformat}
> https://api.travis-ci.org/v3/job/680222168/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13667: [FLINK-19679] Deduplicate code between CheckpointBarrierUnaligner and CheckpointBarrierAligner
flinkbot edited a comment on pull request #13667: URL: https://github.com/apache/flink/pull/13667#issuecomment-710095324 ## CI report: * 5ab532006ccbfb5632b4ad60053203a5e5dea3a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7770) * de848b220414a40d24a3b6ec2656dcbe1ad30f3a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7788) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19687) Support to get execution plan in `StatementSet`
[ https://issues.apache.org/jira/browse/FLINK-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215827#comment-17215827 ] xiaozilong edited comment on FLINK-19687 at 10/17/20, 9:37 AM: --- Hi [~hailong wang], I want to visualize the execution plan in [https://flink.apache.org/visualizer/] tools. The physical execution plan unable to draw. In addition, I think it is allowed to get the execution plan before the job is submitted in Flink SQL 1.11. was (Author: xiaozilong): Hi [~hailong wang], I want to visualize the execution plan in [https://flink.apache.org/visualizer/] tools. The physical execution plan unable to draw. In addition, I think it is allowed to get the execution plan before the job is submitted. > Support to get execution plan in `StatementSet` > --- > > Key: FLINK-19687 > URL: https://issues.apache.org/jira/browse/FLINK-19687 > Project: Flink > Issue Type: Wish > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaozilong >Priority: Major > > Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet > exception "No operators defined in streaming topology. Cannot execute." when > use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I > found translation operations only happen when StatementSet.execute() is > called in Flink SQL 1.11. So we cannot get job's execution plan before the > job submit? Can we support to get execution plan in `StatementSet` or invoke > method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I > think the latter is better so that we can reuse `env.getExecutionPlan()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on a change in pull request #13625: [FLINK-19623][table-planner-blink] Introduce ExecEdge to describe information on input edges for ExecNode
godfreyhe commented on a change in pull request #13625:
URL: https://github.com/apache/flink/pull/13625#discussion_r506906315
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/reuse/DeadlockBreakupProcessor.scala
##
@@ -146,22 +146,33 @@ class DeadlockBreakupProcessor extends DAGProcessor {
class DeadlockBreakupVisitor(finder: ReuseNodeFinder) extends
ExecNodeVisitorImpl {
-private def rewriteJoin(
-join: BatchExecJoinBase,
-leftIsBuild: Boolean,
-distribution: FlinkRelDistribution): Unit = {
- val (buildSideIndex, probeSideIndex) = if (leftIsBuild) (0, 1) else (1,
0)
- val buildNode = join.getInputNodes.get(buildSideIndex)
- val probeNode = join.getInputNodes.get(probeSideIndex)
+private def rewriteTwoInputNode(
+node: ExecNode[_, _],
+leftPriority: Int,
+requiredShuffle: ExecEdge.RequiredShuffle): Unit = {
+ val (buildSideIndex, probeSideIndex) = if (leftPriority == 0) (0, 1)
else (1, 0)
+ val buildNode = node.getInputNodes.get(buildSideIndex)
Review comment:
no build/probe concept here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #13625: [FLINK-19623][table-planner-blink] Introduce ExecEdge to describe information on input edges for ExecNode
godfreyhe commented on a change in pull request #13625:
URL: https://github.com/apache/flink/pull/13625#discussion_r506909706
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/exec/StreamExecNode.scala
##
@@ -21,7 +21,19 @@ package org.apache.flink.table.planner.plan.nodes.exec
import org.apache.flink.table.planner.delegation.StreamPlanner
import org.apache.flink.table.planner.utils.Logging
+import java.util
+
/**
* Base class for stream ExecNode.
*/
-trait StreamExecNode[T] extends ExecNode[StreamPlanner, T] with Logging
+trait StreamExecNode[T] extends ExecNode[StreamPlanner, T] with Logging {
+
+ def getInputEdges: util.List[ExecEdge] = {
+// TODO fill out the required shuffle for each stream exec node
+val edges = new util.ArrayList[ExecEdge]()
+for (_ <- 0 until getInputNodes.size()) {
+ edges.add(ExecEdge.DEFAULT)
+}
+edges
Review comment:
nit: can be simplified as `getInputNodes.map(_ => ExecEdge.DEFAULT)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #13625: [FLINK-19623][table-planner-blink] Introduce ExecEdge to describe information on input edges for ExecNode
godfreyhe commented on a change in pull request #13625:
URL: https://github.com/apache/flink/pull/13625#discussion_r506903320
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/exec/BatchExecNode.scala
##
@@ -25,11 +25,4 @@ import org.apache.flink.table.planner.utils.Logging
/**
* Base class for batch ExecNode.
*/
-trait BatchExecNode[T] extends ExecNode[BatchPlanner, T] with Logging {
-
- /**
-* Returns [[DamBehavior]] of this node.
-*/
- def getDamBehavior: DamBehavior
-
-}
+trait BatchExecNode[T] extends ExecNode[BatchPlanner, T] with Logging
Review comment:
nit: remove unused imports
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/batch/BatchExecUnion.scala
##
@@ -98,11 +98,11 @@ class BatchExecUnion(
//~ ExecNode methods
---
- override def getDamBehavior: DamBehavior = DamBehavior.PIPELINED
-
override def getInputNodes: util.List[ExecNode[BatchPlanner, _]] =
getInputs.map(_.asInstanceOf[ExecNode[BatchPlanner, _]])
+ override def getInputEdges: util.List[ExecEdge] = List(ExecEdge.DEFAULT,
ExecEdge.DEFAULT)
Review comment:
may have more than two inputs
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/exec/StreamExecNode.scala
##
@@ -21,7 +21,19 @@ package org.apache.flink.table.planner.plan.nodes.exec
import org.apache.flink.table.planner.delegation.StreamPlanner
import org.apache.flink.table.planner.utils.Logging
+import java.util
+
/**
* Base class for stream ExecNode.
*/
-trait StreamExecNode[T] extends ExecNode[StreamPlanner, T] with Logging
+trait StreamExecNode[T] extends ExecNode[StreamPlanner, T] with Logging {
+
+ def getInputEdges: util.List[ExecEdge] = {
+// TODO fill out the required shuffle for each stream exec node
+val edges = new util.ArrayList[ExecEdge]()
+for (_ <- 0 until getInputNodes.size()) {
+ edges.add(ExecEdge.DEFAULT)
+}
+edges
Review comment:
can be simplified as `getInputNodes.map(_ => ExecEdge.DEFAULT)`
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/reuse/DeadlockBreakupProcessor.scala
##
@@ -146,22 +146,33 @@ class DeadlockBreakupProcessor extends DAGProcessor {
class DeadlockBreakupVisitor(finder: ReuseNodeFinder) extends
ExecNodeVisitorImpl {
-private def rewriteJoin(
-join: BatchExecJoinBase,
-leftIsBuild: Boolean,
-distribution: FlinkRelDistribution): Unit = {
- val (buildSideIndex, probeSideIndex) = if (leftIsBuild) (0, 1) else (1,
0)
- val buildNode = join.getInputNodes.get(buildSideIndex)
- val probeNode = join.getInputNodes.get(probeSideIndex)
+private def rewriteTwoInputNode(
+node: ExecNode[_, _],
+leftPriority: Int,
+requiredShuffle: ExecEdge.RequiredShuffle): Unit = {
+ val (buildSideIndex, probeSideIndex) = if (leftPriority == 0) (0, 1)
else (1, 0)
+ val buildNode = node.getInputNodes.get(buildSideIndex)
Review comment:
not build/probe concept here
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/reuse/DeadlockBreakupProcessor.scala
##
@@ -320,20 +334,24 @@ class DeadlockBreakupProcessor extends DAGProcessor {
// should exclude the reused node (at last position in path)
while (!hasFullDamNode && idx < inputPath.length - 1) {
val node = inputPath(idx)
- val nodeDamBehavior =
node.asInstanceOf[BatchExecNode[_]].getDamBehavior
- hasFullDamNode = if (nodeDamBehavior == DamBehavior.FULL_DAM) {
+ val atLeastEndInput = node.getInputEdges.forall(
+e =>
e.getDamBehavior.stricterOrEqual(ExecEdge.DamBehavior.END_INPUT))
+ hasFullDamNode = if (atLeastEndInput) {
true
} else {
-node match {
- case h: BatchExecHashJoin =>
-val buildSideIndex = if (h.leftIsBuild) 0 else 1
-val buildNode = h.getInputNodes.get(buildSideIndex)
-checkJoinBuildSide(buildNode, idx, inputPath)
- case n: BatchExecNestedLoopJoin =>
-val buildSideIndex = if (n.leftIsBuild) 0 else 1
-val buildNode = n.getInputNodes.get(buildSideIndex)
-checkJoinBuildSide(buildNode, idx, inputPath)
- case _ => false
+val inputEdges = node.getInputEdges
+if (inputEdges.size() == 2) {
+ val leftPriority = inputEdges.get(0).getPriority
+ val rightPriority = inputEdges.get(1).getPriority
+ if (leftPriority != rightPriority) {
+val buildSideIndex = if (leftPriority ==
[GitHub] [flink] godfreyhe commented on a change in pull request #13625: [FLINK-19623][table-planner-blink] Introduce ExecEdge to describe information on input edges for ExecNode
godfreyhe commented on a change in pull request #13625:
URL: https://github.com/apache/flink/pull/13625#discussion_r506906883
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/reuse/DeadlockBreakupProcessor.scala
##
@@ -146,22 +146,33 @@ class DeadlockBreakupProcessor extends DAGProcessor {
class DeadlockBreakupVisitor(finder: ReuseNodeFinder) extends
ExecNodeVisitorImpl {
-private def rewriteJoin(
-join: BatchExecJoinBase,
-leftIsBuild: Boolean,
-distribution: FlinkRelDistribution): Unit = {
- val (buildSideIndex, probeSideIndex) = if (leftIsBuild) (0, 1) else (1,
0)
- val buildNode = join.getInputNodes.get(buildSideIndex)
- val probeNode = join.getInputNodes.get(probeSideIndex)
+private def rewriteTwoInputNode(
+node: ExecNode[_, _],
+leftPriority: Int,
+requiredShuffle: ExecEdge.RequiredShuffle): Unit = {
+ val (buildSideIndex, probeSideIndex) = if (leftPriority == 0) (0, 1)
else (1, 0)
Review comment:
the highest priority is always 0? do we have this guarantee?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13668: [hotfix] Log interrupted exception on debug when closing threads in ExternalSorter
rmetzger commented on a change in pull request #13668:
URL: https://github.com/apache/flink/pull/13668#discussion_r506908873
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/ExternalSorter.java
##
@@ -230,6 +218,20 @@ public void close() {
}
}
+ private void closeThread(StageRunner sortThread, String threadName) {
Review comment:
```suggestion
private void closeThread(StageRunner thread, String threadName) {
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/ExternalSorter.java
##
@@ -230,6 +218,20 @@ public void close() {
}
}
+ private void closeThread(StageRunner sortThread, String threadName) {
+ try {
+ sortThread.close();
Review comment:
```suggestion
thread.close();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-19688) Flink job gets into restart loop caused by InterruptedExceptions
[
https://issues.apache.org/jira/browse/FLINK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-19688:
---
Fix Version/s: 1.12.0
> Flink job gets into restart loop caused by InterruptedExceptions
>
>
> Key: FLINK-19688
> URL: https://issues.apache.org/jira/browse/FLINK-19688
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Fix For: 1.12.0
>
> Attachments: logs.tgz
>
>
> I have a benchmarking test job, that throws RuntimeExceptions at any operator
> at a configured, random interval. When using low intervals, such as mean
> failure rate = 60 s, the job will get into a state where it frequently fails
> with InterruptedExceptions.
> The same job does not have this problem on Flink 1.11.2 (at least not after
> running the job for 15 hours, on 1.12-SN, it happens within a few minutes)
> This is the job:
> https://github.com/rmetzger/flip1-bench/blob/master/flip1-bench-jobs/src/main/java/com/ververica/TPCHQuery3.java
> This is the exception:
> {code}
> 2020-10-16 16:02:15,653 WARN org.apache.flink.runtime.taskmanager.Task
> [] - CHAIN GroupReduce (GroupReduce at
> main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38)) (8/8)#1
> (06d656f696bf4ed98831938a7ac2359d_c1c4a56fea0536703d37867c057f0cc8_7_1)
> switched from RUNNING to FAILED.
> java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
> (GroupReduce at main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38))' , caused an error:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:370)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> Caused by: org.apache.flink.util.WrappingRuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:253)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error obtaining the sorted input: Thread
> 'SortMerger Reading Thread' terminated due to an exception: Connection for
> partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> ~[?:1.8.0_222]
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> ~[?:1.8.0_222]
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:250)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by:
[jira] [Updated] (FLINK-19688) Flink job gets into restart loop caused by InterruptedExceptions
[
https://issues.apache.org/jira/browse/FLINK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-19688:
---
Attachment: logs.tgz
> Flink job gets into restart loop caused by InterruptedExceptions
>
>
> Key: FLINK-19688
> URL: https://issues.apache.org/jira/browse/FLINK-19688
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Attachments: logs.tgz
>
>
> I have a benchmarking test job, that throws RuntimeExceptions at any operator
> at a configured, random interval. When using low intervals, such as mean
> failure rate = 60 s, the job will get into a state where it frequently fails
> with InterruptedExceptions.
> The same job does not have this problem on Flink 1.11.2 (at least not after
> running the job for 15 hours, on 1.12-SN, it happens within a few minutes)
> This is the job:
> https://github.com/rmetzger/flip1-bench/blob/master/flip1-bench-jobs/src/main/java/com/ververica/TPCHQuery3.java
> This is the exception:
> {code}
> 2020-10-16 16:02:15,653 WARN org.apache.flink.runtime.taskmanager.Task
> [] - CHAIN GroupReduce (GroupReduce at
> main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38)) (8/8)#1
> (06d656f696bf4ed98831938a7ac2359d_c1c4a56fea0536703d37867c057f0cc8_7_1)
> switched from RUNNING to FAILED.
> java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
> (GroupReduce at main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38))' , caused an error:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:370)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> Caused by: org.apache.flink.util.WrappingRuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:253)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error obtaining the sorted input: Thread
> 'SortMerger Reading Thread' terminated due to an exception: Connection for
> partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> ~[?:1.8.0_222]
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> ~[?:1.8.0_222]
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:250)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by: java.lang.RuntimeException: Error
[jira] [Updated] (FLINK-19688) Flink job gets into restart loop caused by InterruptedExceptions
[
https://issues.apache.org/jira/browse/FLINK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-19688:
---
Priority: Critical (was: Major)
> Flink job gets into restart loop caused by InterruptedExceptions
>
>
> Key: FLINK-19688
> URL: https://issues.apache.org/jira/browse/FLINK-19688
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
>
> I have a benchmarking test job, that throws RuntimeExceptions at any operator
> at a configured, random interval. When using low intervals, such as mean
> failure rate = 60 s, the job will get into a state where it frequently fails
> with InterruptedExceptions.
> The same job does not have this problem on Flink 1.11.2 (at least not after
> running the job for 15 hours, on 1.12-SN, it happens within a few minutes)
> This is the job:
> https://github.com/rmetzger/flip1-bench/blob/master/flip1-bench-jobs/src/main/java/com/ververica/TPCHQuery3.java
> This is the exception:
> {code}
> 2020-10-16 16:02:15,653 WARN org.apache.flink.runtime.taskmanager.Task
> [] - CHAIN GroupReduce (GroupReduce at
> main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38)) (8/8)#1
> (06d656f696bf4ed98831938a7ac2359d_c1c4a56fea0536703d37867c057f0cc8_7_1)
> switched from RUNNING to FAILED.
> java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
> (GroupReduce at main(TPCHQuery3.java:199)) -> Map (Map at
> appendMapper(KillerClientMapper.java:38))' , caused an error:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:370)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> Caused by: org.apache.flink.util.WrappingRuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
> to an exception: Connection for partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:253)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error obtaining the sorted input: Thread
> 'SortMerger Reading Thread' terminated due to an exception: Connection for
> partition
> 060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
> not reachable.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> ~[?:1.8.0_222]
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> ~[?:1.8.0_222]
> at
> org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:250)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> ... 4 more
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
[jira] [Created] (FLINK-19688) Flink job gets into restart loop caused by InterruptedExceptions
Robert Metzger created FLINK-19688:
--
Summary: Flink job gets into restart loop caused by
InterruptedExceptions
Key: FLINK-19688
URL: https://issues.apache.org/jira/browse/FLINK-19688
Project: Flink
Issue Type: Bug
Components: Runtime / Network, Runtime / Task
Affects Versions: 1.12.0
Reporter: Robert Metzger
I have a benchmarking test job, that throws RuntimeExceptions at any operator
at a configured, random interval. When using low intervals, such as mean
failure rate = 60 s, the job will get into a state where it frequently fails
with InterruptedExceptions.
The same job does not have this problem on Flink 1.11.2 (at least not after
running the job for 15 hours, on 1.12-SN, it happens within a few minutes)
This is the job:
https://github.com/rmetzger/flip1-bench/blob/master/flip1-bench-jobs/src/main/java/com/ververica/TPCHQuery3.java
This is the exception:
{code}
2020-10-16 16:02:15,653 WARN org.apache.flink.runtime.taskmanager.Task
[] - CHAIN GroupReduce (GroupReduce at main(TPCHQuery3.java:199)) ->
Map (Map at appendMapper(KillerClientMapper.java:38)) (8/8)#1
(06d656f696bf4ed98831938a7ac2359d_c1c4a56fea0536703d37867c057f0cc8_7_1)
switched from RUNNING to FAILED.
java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
(GroupReduce at main(TPCHQuery3.java:199)) -> Map (Map at
appendMapper(KillerClientMapper.java:38))' , caused an error:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
to an exception: Connection for partition
060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
not reachable.
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:370)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: org.apache.flink.util.WrappingRuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
to an exception: Connection for partition
060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
not reachable.
at
org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:253)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
... 4 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated
due to an exception: Connection for partition
060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
not reachable.
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_222]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
~[?:1.8.0_222]
at
org.apache.flink.runtime.operators.sort.ExternalSorter.getIterator(ExternalSorter.java:250)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1122)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:475)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
... 4 more
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread
'SortMerger Reading Thread' terminated due to an exception: Connection for
partition
060d457c4163472f65a4b741993c83f8#0@06d656f696bf4ed98831938a7ac2359d_0bcc9fbf9ac242d5aac540917d980e44_0_1
not reachable.
at
org.apache.flink.runtime.operators.sort.ExternalSorter.lambda$getIterator$1(ExternalSorter.java:247)
[jira] [Commented] (FLINK-19687) Support to get execution plan in `StatementSet`
[ https://issues.apache.org/jira/browse/FLINK-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215827#comment-17215827 ] xiaozilong commented on FLINK-19687: Hi [~hailong wang], I want to visualize the execution plan in [https://flink.apache.org/visualizer/] tools. The physical execution plan unable to draw. In addition, I think it is allowed to get the execution plan before the job is submitted. > Support to get execution plan in `StatementSet` > --- > > Key: FLINK-19687 > URL: https://issues.apache.org/jira/browse/FLINK-19687 > Project: Flink > Issue Type: Wish > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaozilong >Priority: Major > > Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet > exception "No operators defined in streaming topology. Cannot execute." when > use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I > found translation operations only happen when StatementSet.execute() is > called in Flink SQL 1.11. So we cannot get job's execution plan before the > job submit? Can we support to get execution plan in `StatementSet` or invoke > method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I > think the latter is better so that we can reuse `env.getExecutionPlan()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13667: [FLINK-19679] Deduplicate code between CheckpointBarrierUnaligner and CheckpointBarrierAligner
flinkbot edited a comment on pull request #13667: URL: https://github.com/apache/flink/pull/13667#issuecomment-710095324 ## CI report: * 5ab532006ccbfb5632b4ad60053203a5e5dea3a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7770) * de848b220414a40d24a3b6ec2656dcbe1ad30f3a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13671: [FLINK-19625][table-planner] Introduce multi-input exec node
flinkbot edited a comment on pull request #13671: URL: https://github.com/apache/flink/pull/13671#issuecomment-710779251 ## CI report: * 745d079d12088ca1faeb89a7b7974c8bbea383b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7787) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13671: [FLINK-19625][table-planner] Introduce multi-input exec node
flinkbot commented on pull request #13671: URL: https://github.com/apache/flink/pull/13671#issuecomment-710779251 ## CI report: * 745d079d12088ca1faeb89a7b7974c8bbea383b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13643: [Hotfix][CheckStyle][Min] Fix DummyInputFormat class lacks Javadoc
flinkbot edited a comment on pull request #13643: URL: https://github.com/apache/flink/pull/13643#issuecomment-708870329 ## CI report: * Unknown: [CANCELED](TBD) * 5fe6724b54691b8283fb6ddce689d53a8a420fda Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7786) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13669: [FLINK-19684][Connector][jdbc] Fix the Jdbc-connector's 'lookup.max-retries' option implementation
wangxlong commented on pull request #13669: URL: https://github.com/apache/flink/pull/13669#issuecomment-710778642 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19687) Support to get execution plan in `StatementSet`
[ https://issues.apache.org/jira/browse/FLINK-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215819#comment-17215819 ] hailong wang commented on FLINK-19687: -- Hi [~xiaozilong], the physical execution plan introduced by StatementSet#explain is what you want? IMO, it is no necessary to introduce that again. > Support to get execution plan in `StatementSet` > --- > > Key: FLINK-19687 > URL: https://issues.apache.org/jira/browse/FLINK-19687 > Project: Flink > Issue Type: Wish > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaozilong >Priority: Major > > Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet > exception "No operators defined in streaming topology. Cannot execute." when > use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I > found translation operations only happen when StatementSet.execute() is > called in Flink SQL 1.11. So we cannot get job's execution plan before the > job submit? Can we support to get execution plan in `StatementSet` or invoke > method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I > think the latter is better so that we can reuse `env.getExecutionPlan()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13670: [hotfix][docs]Fix typos in docs
flinkbot edited a comment on pull request #13670: URL: https://github.com/apache/flink/pull/13670#issuecomment-710731909 ## CI report: * eaf3ff6217b7306a08a893ffd154047488cc57bb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7782) * 66c0f3542d0f0898a2eba8013ab4a390f35c263d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7785) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13643: [Hotfix][CheckStyle][Min] Fix DummyInputFormat class lacks Javadoc
flinkbot edited a comment on pull request #13643: URL: https://github.com/apache/flink/pull/13643#issuecomment-708870329 ## CI report: * Unknown: [CANCELED](TBD) * 5fe6724b54691b8283fb6ddce689d53a8a420fda UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13602: [FLINK-19346][coordination] Generate and put ClusterPartitionDescriptor of ClusterPartition in JobResult when job finishes
flinkbot edited a comment on pull request #13602: URL: https://github.com/apache/flink/pull/13602#issuecomment-70743 ## CI report: * 5bd17fad8f85ce110218ef2ec98912b138db03e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7784) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7763) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13671: [FLINK-19625][table-planner] Introduce multi-input exec node
flinkbot commented on pull request #13671: URL: https://github.com/apache/flink/pull/13671#issuecomment-710775846 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 7cc5e9cc446d5c9d62fc733dc413550fbf4be866 (Sat Oct 17 08:34:40 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe opened a new pull request #13671: [FLINK-19625][table-planner] Introduce multi-input exec node
godfreyhe opened a new pull request #13671: URL: https://github.com/apache/flink/pull/13671 ## What is the purpose of the change *This pr aims to introduce multi-input exec node for blink planner. Currently, all the exec nodes are extends from both FlinkPhysicalRel and ExecNode, because the ExecNode#translateToPlan method will access some info of FlinkPhysicalRel, For minimal changes, we also let multi-input exec node extend them as a temporary solution* ## Brief change log - *Introduce BatchExecMultipleInputNode and StreamExecMultipleInputNode, and implement the necessary methods* ## Verifying this change This change added tests and can be verified as follows: - *Extended ExplainTest to verify the explain result of multiple input plan* - *Extended FlinkRelMdUniqueKeysTest to verify the result of `getUniqueKeys` on multiple input rel * ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19625) Introduce multi-input exec node
[ https://issues.apache.org/jira/browse/FLINK-19625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19625: --- Labels: pull-request-available (was: ) > Introduce multi-input exec node > --- > > Key: FLINK-19625 > URL: https://issues.apache.org/jira/browse/FLINK-19625 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Caizhi Weng >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > For multi-input to work in Blink planner, we should first introduce > multi-input exec node in the planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangxlong commented on pull request #13643: [Hotfix][CheckStyle][Min] Fix DummyInputFormat class lacks Javadoc
wangxlong commented on pull request #13643: URL: https://github.com/apache/flink/pull/13643#issuecomment-710774486 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13670: [hotfix][docs]Fix typos in docs
flinkbot edited a comment on pull request #13670: URL: https://github.com/apache/flink/pull/13670#issuecomment-710731909 ## CI report: * eaf3ff6217b7306a08a893ffd154047488cc57bb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7782) * 66c0f3542d0f0898a2eba8013ab4a390f35c263d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13602: [FLINK-19346][coordination] Generate and put ClusterPartitionDescriptor of ClusterPartition in JobResult when job finishes
flinkbot edited a comment on pull request #13602: URL: https://github.com/apache/flink/pull/13602#issuecomment-70743 ## CI report: * 5bd17fad8f85ce110218ef2ec98912b138db03e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7763) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7784) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19530) Table Concepts Page
[ https://issues.apache.org/jira/browse/FLINK-19530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215813#comment-17215813 ] Kartik Khare commented on FLINK-19530: -- Some queries - * Written a brief info on Planner but is there a design document or something similar which can help me to take a deep dive into planners. * There is already a page for data types in Type Serialization. Should some of that info be transferred here? > Table Concepts Page > --- > > Key: FLINK-19530 > URL: https://issues.apache.org/jira/browse/FLINK-19530 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Seth Wiesman >Assignee: Kartik Khare >Priority: Major > > Concepts > What are the general concepts (independent of API/SQL Client) any user should > know about? > We put this at the end and link from the main pages to pages here if > necessary. > Planners > What is a Planner? Temporary docs. Removed in the future. > Blink Planner Features and Limitations > Flink Planner Features and Limitations > Data Types > Which data can we process? > Unbounded Data Processing > Which operation needs special attention when working with unbounded data? > Dynamic Tables (with all update modes) > Time Attributes > Query Configuration > Joins in Continuous Queries > Temporal Tables > Explain the concept of a temporal table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Sxnan commented on pull request #13602: [FLINK-19346][coordination] Generate and put ClusterPartitionDescriptor of ClusterPartition in JobResult when job finishes
Sxnan commented on pull request #13602: URL: https://github.com/apache/flink/pull/13602#issuecomment-710770590 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19687) Support to get execution plan in `StatementSet`
[ https://issues.apache.org/jira/browse/FLINK-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215811#comment-17215811 ] xiaozilong commented on FLINK-19687: [~jark] Hi Jark, can you help me to review this issue? > Support to get execution plan in `StatementSet` > --- > > Key: FLINK-19687 > URL: https://issues.apache.org/jira/browse/FLINK-19687 > Project: Flink > Issue Type: Wish > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaozilong >Priority: Major > > Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet > exception "No operators defined in streaming topology. Cannot execute." when > use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I > found translation operations only happen when StatementSet.execute() is > called in Flink SQL 1.11. So we cannot get job's execution plan before the > job submit? Can we support to get execution plan in `StatementSet` or invoke > method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I > think the latter is better so that we can reuse `env.getExecutionPlan()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19687) Support to get execution plan in `StatementSet`
xiaozilong created FLINK-19687: -- Summary: Support to get execution plan in `StatementSet` Key: FLINK-19687 URL: https://issues.apache.org/jira/browse/FLINK-19687 Project: Flink Issue Type: Wish Components: Table SQL / API Affects Versions: 1.11.0 Reporter: xiaozilong Hi, I want to get job's execution plan in Flink SQL 1.11, but i meet exception "No operators defined in streaming topology. Cannot execute." when use `env.getExecutionPlan()`. The same code runs fine in Flink SQL 1.10. I found translation operations only happen when StatementSet.execute() is called in Flink SQL 1.11. So we cannot get job's execution plan before the job submit? Can we support to get execution plan in `StatementSet` or invoke method `TableEnvironmentImpl#translate` in `StatementSetImpl#addInsertSql`? I think the latter is better so that we can reuse `env.getExecutionPlan()`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover
[
https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215808#comment-17215808
]
Nicholas Jiang edited comment on FLINK-19596 at 10/17/20, 7:44 AM:
---
[~wind_ljy], I agree with the point you mentioned. CheckpointCoordinator could
restore latest checkpointed state in the cases including local or regional
failover, global restore style operation and restoring the state with the
savepoint. CheckpointCoordinator doesn't need to recover the checkpoints on
each recovery of local or regional failover because this failover lack of the
leadership. We could add the isGlobalRecovery check for
completedCheckpointStore.recover caller.
cc [~trohrmann],[~zjwang]
was (Author: nicholasjiang):
[~wind_ljy], I agree with the point you mentioned. CheckpointCoordinator could
restore latest checkpointed state in the cases including local or regional
failover, global restore style operation and restoring the state with the
savepoint. CheckpointCoordinator doesn't need to recover the checkpoints on
each recovery of local or regional failover because this failover lack of the
leadership. We could add the isGlobalRecovery check for
completedCheckpointStore.recover caller.
cc [~trohrmann][~zjwang]
> Do not recover CompletedCheckpointStore on each failover
>
>
> Key: FLINK-19596
> URL: https://issues.apache.org/jira/browse/FLINK-19596
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.2
>Reporter: Jiayi Liao
>Priority: Major
>
> {{completedCheckpointStore.recover()}} in
> {{restoreLatestCheckpointedStateInternal}} could be a bottleneck on failover
> because the {{CompletedCheckpointStore}} needs to load HDFS files to
> instantialize the {{CompleteCheckpoint}} instances.
> The impact is significant in our case below:
> * Jobs with high parallelism (no shuffle) which transfer data from Kafka to
> other filesystems.
> * If a machine goes down, several containers and tens of tasks are affected,
> which means the {{completedCheckpointStore.recover()}} would be called tens
> of times since the tasks are not in a failover region.
> And I notice there is a "TODO" in the source codes:
> {code:java}
> // Recover the checkpoints, TODO this could be done only when there is a new
> leader, not on each recovery
> completedCheckpointStore.recover();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover
[
https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215808#comment-17215808
]
Nicholas Jiang commented on FLINK-19596:
[~wind_ljy], I agree with the point you mentioned. CheckpointCoordinator could
restore latest checkpointed state in the cases including local or regional
failover, global restore style operation and restoring the state with the
savepoint. CheckpointCoordinator doesn't need to recover the checkpoints on
each recovery of local or regional failover because this failover lack of the
leadership. We could add the isGlobalRecovery check for
completedCheckpointStore.recover caller.
cc [~trohrmann]
> Do not recover CompletedCheckpointStore on each failover
>
>
> Key: FLINK-19596
> URL: https://issues.apache.org/jira/browse/FLINK-19596
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.2
>Reporter: Jiayi Liao
>Priority: Major
>
> {{completedCheckpointStore.recover()}} in
> {{restoreLatestCheckpointedStateInternal}} could be a bottleneck on failover
> because the {{CompletedCheckpointStore}} needs to load HDFS files to
> instantialize the {{CompleteCheckpoint}} instances.
> The impact is significant in our case below:
> * Jobs with high parallelism (no shuffle) which transfer data from Kafka to
> other filesystems.
> * If a machine goes down, several containers and tens of tasks are affected,
> which means the {{completedCheckpointStore.recover()}} would be called tens
> of times since the tasks are not in a failover region.
> And I notice there is a "TODO" in the source codes:
> {code:java}
> // Recover the checkpoints, TODO this could be done only when there is a new
> leader, not on each recovery
> completedCheckpointStore.recover();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover
[
https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215808#comment-17215808
]
Nicholas Jiang edited comment on FLINK-19596 at 10/17/20, 7:43 AM:
---
[~wind_ljy], I agree with the point you mentioned. CheckpointCoordinator could
restore latest checkpointed state in the cases including local or regional
failover, global restore style operation and restoring the state with the
savepoint. CheckpointCoordinator doesn't need to recover the checkpoints on
each recovery of local or regional failover because this failover lack of the
leadership. We could add the isGlobalRecovery check for
completedCheckpointStore.recover caller.
cc [~trohrmann][~zjwang]
was (Author: nicholasjiang):
[~wind_ljy], I agree with the point you mentioned. CheckpointCoordinator could
restore latest checkpointed state in the cases including local or regional
failover, global restore style operation and restoring the state with the
savepoint. CheckpointCoordinator doesn't need to recover the checkpoints on
each recovery of local or regional failover because this failover lack of the
leadership. We could add the isGlobalRecovery check for
completedCheckpointStore.recover caller.
cc [~trohrmann]
> Do not recover CompletedCheckpointStore on each failover
>
>
> Key: FLINK-19596
> URL: https://issues.apache.org/jira/browse/FLINK-19596
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.2
>Reporter: Jiayi Liao
>Priority: Major
>
> {{completedCheckpointStore.recover()}} in
> {{restoreLatestCheckpointedStateInternal}} could be a bottleneck on failover
> because the {{CompletedCheckpointStore}} needs to load HDFS files to
> instantialize the {{CompleteCheckpoint}} instances.
> The impact is significant in our case below:
> * Jobs with high parallelism (no shuffle) which transfer data from Kafka to
> other filesystems.
> * If a machine goes down, several containers and tens of tasks are affected,
> which means the {{completedCheckpointStore.recover()}} would be called tens
> of times since the tasks are not in a failover region.
> And I notice there is a "TODO" in the source codes:
> {code:java}
> // Recover the checkpoints, TODO this could be done only when there is a new
> leader, not on each recovery
> completedCheckpointStore.recover();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13669: [FLINK-19684][Connector][jdbc] Fix the Jdbc-connector's 'lookup.max-retries' option implementation
flinkbot edited a comment on pull request #13669: URL: https://github.com/apache/flink/pull/13669#issuecomment-710232578 ## CI report: * 54f00eb50174658ded96de3473d452d07caaca99 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7783) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13657: [FLINK-19669][coordination] PipelinedRegionSchedulingStrategy#init ResultPartitionType blocking check use isBlocking method
flinkbot edited a comment on pull request #13657: URL: https://github.com/apache/flink/pull/13657#issuecomment-709726515 ## CI report: * 7461070175526095fd5741748f413ba4f78246d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7781) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7731) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
