[jira] [Commented] (FLINK-14815) Expose network metric for sub task in rest api
[ https://issues.apache.org/jira/browse/FLINK-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195291#comment-17195291 ] lining commented on FLINK-14815: It is still valid. > Expose network metric for sub task in rest api > -- > > Key: FLINK-14815 > URL: https://issues.apache.org/jira/browse/FLINK-14815 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / Network, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > * SubTask > ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, > inputFloatingBuffersUsage. > *** If the subtask is not back pressured, but it is causing backpressure > (full input, empty output) > *** By comparing exclusive/floating buffers usage, whether all channels are > back-pressure or only some of them > ** back-pressured for show whether it is back pressured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14435) Add TaskManageResourceInfo which match the memory compositions of taskmanager
[
https://issues.apache.org/jira/browse/FLINK-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195285#comment-17195285
]
lining commented on FLINK-14435:
The previous configuration doesn't fully reflect the memory model of the task
manager. If the user chooses to configure the task manager's total memory, he
doesn't know how Flink allocates memory to heap/native/direct memory
internally. But if the user chooses to specify each part and the total
explicitly, then there's a possibility that the sum up of the part doesn't
match to the whole, then the job won't be started normally.
> Add TaskManageResourceInfo which match the memory compositions of taskmanager
> -
>
> Key: FLINK-14435
> URL: https://issues.apache.org/jira/browse/FLINK-14435
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: Matthias
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> * information from TaskExecutorResourceSpec in flip-49, add it to
> TaskExecutorRegistration.
> {code:json}
> public class TaskManagerResourceInfo {
> private final double cpuCores;
> private final long frameworkHeap;
> private final long frameworkOffHeap;
> private final long taskHeap;
> private final long taskOffHeap;
> private final long shuffleMemory;
> private final long managedMemory;
> private final long jvmMetaSpace;
> private final long jvmOverhead;
> private final long totalProcessMemory;
> }{code}
> * url: /taskmanagers/:taskmanagerid
> * response: add
> {code:java}
> resource: {
> cpuCores: 4,
> frameworkHeap: 134217728,
> frameworkOffHeap: 134217728,
> taskHeap: 181193928,
> taskOffHeap: 0,
> shuffleMemory: 33554432,
> managedMemory: 322122552,
> jvmMetaSpace: 134217728,
> jvmOverhead: 134217728,
> totalProcessMemory: 1073741824
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195262#comment-17195262 ] lining commented on FLINK-17328: WebUI has monitor backpressure. But users need to know current and upstream's network metric to judge current whether is the source of backpressure. Now users have to record relevant information. It is just improved for the old function. > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > Labels: pull-request-available > > JobDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15170) WebFrontendITCase.testCancelYarn fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17174264#comment-17174264
]
lining commented on FLINK-15170:
According to the current message, it can be determined that this is due to
{quote}client.sendGetRequest("/jobs/" + jid + "/yarn-cancel",
getTimeLeft(deadline));
{quote}
timeout.
> WebFrontendITCase.testCancelYarn fails on travis
>
>
> Key: FLINK-15170
> URL: https://issues.apache.org/jira/browse/FLINK-15170
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Web Frontend
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: lining
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0, 1.11.2, 1.10.3
>
>
> https://api.travis-ci.org/v3/job/622756846/log.txt
> {code}
> 23:59:33.299 [INFO] Running
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:33.365 [INFO] Running
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:35.379 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 2.012 s - in
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:38.802 [ERROR] Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time
> elapsed: 5.5 s <<< FAILURE! - in
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:38.802 [ERROR]
> testCancelYarn(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.436 s <<< ERROR!
> java.util.concurrent.TimeoutException: Connection failed
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancelYarn(WebFrontendITCase.java:324)
> 23:59:38.802 [ERROR]
> testCancel(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.003 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancel(WebFrontendITCase.java:240)
> 23:59:39.193 [INFO]
> 23:59:39.193 [INFO] Results:
> 23:59:39.193 [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18347) kinesis connector throw Error java.lang.NoSuchFieldError: NO_INTS
[ https://issues.apache.org/jira/browse/FLINK-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-18347: --- Summary: kinesis connector throw Error java.lang.NoSuchFieldError: NO_INTS (was: Error java.lang.NoSuchFieldError: NO_INTS) > kinesis connector throw Error java.lang.NoSuchFieldError: NO_INTS > - > > Key: FLINK-18347 > URL: https://issues.apache.org/jira/browse/FLINK-18347 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.10.1 >Reporter: lining >Priority: Major > > java.lang.NoSuchFieldError: NO_INTSjava.lang.NoSuchFieldError: NO_INTS at > com.fasterxml.jackson.dataformat.cbor.CBORParser.(CBORParser.java:285) > ~[usercode.jar:?] at > com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.constructParser(CBORParserBootstrapper.java:91) > ~[usercode.jar:?] at > com.fasterxml.jackson.dataformat.cbor.CBORFactory._createParser(CBORFactory.java:399) > ~[usercode.jar:?] at > com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:324) > ~[usercode.jar:?] at > com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:26) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:109) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:43) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1627) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1336) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2809) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2765) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeListShards(AmazonKinesisClient.java:1557) > ~[usercode.jar:?] at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.listShards(AmazonKinesisClient.java:1528) > ~[usercode.jar:?] at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.listShards(KinesisProxy.java:439) > ~[usercode.jar:?] at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:389) > ~[usercode.jar:?] at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:279) > ~[usercode.jar:?] at > org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:686) > ~[usercode.jar:?] at > org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:287) > ~[usercode.jar:?] at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) > ~[flink-dist_2.11-1.10-vvr-1.0.2-SNAPSHOT.jar:1.10-vvr-1.0.2-SNAPSHOT] at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) >
[jira] [Created] (FLINK-18347) Error java.lang.NoSuchFieldError: NO_INTS
lining created FLINK-18347: -- Summary: Error java.lang.NoSuchFieldError: NO_INTS Key: FLINK-18347 URL: https://issues.apache.org/jira/browse/FLINK-18347 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.10.1 Reporter: lining java.lang.NoSuchFieldError: NO_INTSjava.lang.NoSuchFieldError: NO_INTS at com.fasterxml.jackson.dataformat.cbor.CBORParser.(CBORParser.java:285) ~[usercode.jar:?] at com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.constructParser(CBORParserBootstrapper.java:91) ~[usercode.jar:?] at com.fasterxml.jackson.dataformat.cbor.CBORFactory._createParser(CBORFactory.java:399) ~[usercode.jar:?] at com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:324) ~[usercode.jar:?] at com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:26) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:109) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:43) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1627) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1336) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2809) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2765) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeListShards(AmazonKinesisClient.java:1557) ~[usercode.jar:?] at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.listShards(AmazonKinesisClient.java:1528) ~[usercode.jar:?] at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.listShards(KinesisProxy.java:439) ~[usercode.jar:?] at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardsOfStream(KinesisProxy.java:389) ~[usercode.jar:?] at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardList(KinesisProxy.java:279) ~[usercode.jar:?] at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.discoverNewShardsToSubscribe(KinesisDataFetcher.java:686) ~[usercode.jar:?] at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(FlinkKinesisConsumer.java:287) ~[usercode.jar:?] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.11-1.10-vvr-1.0.2-SNAPSHOT.jar:1.10-vvr-1.0.2-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.11-1.10-vvr-1.0.2-SNAPSHOT.jar:1.10-vvr-1.0.2-SNAPSHOT] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200) ~[flink-dist_2.11-1.10-vvr-1.0.2-SNAPSHOT.jar:1.10-vvr-1.0.2-SNAPSHOT] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17378) KafkaProducerExactlyOnceITCase>KafkaProducerTestBase.testExactlyOnceCustomOperator unstable
[
https://issues.apache.org/jira/browse/FLINK-17378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110852#comment-17110852
]
lining commented on FLINK-17378:
Another instance:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1256=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20]
> KafkaProducerExactlyOnceITCase>KafkaProducerTestBase.testExactlyOnceCustomOperator
> unstable
> ---
>
> Key: FLINK-17378
> URL: https://issues.apache.org/jira/browse/FLINK-17378
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> CI run:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=221=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-04-25T00:41:01.4191956Z 00:41:01,418 [Source: Custom Source -> Map ->
> Sink: Unnamed (1/1)] INFO
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
> [] - Flushing new partitions
> 2020-04-25T00:41:01.4194268Z 00:41:01,418 [FailingIdentityMapper Status
> Printer] INFO
> org.apache.flink.streaming.connectors.kafka.testutils.FailingIdentityMapper
> [] - > Failing mapper 0: count=690,
> totalCount=1000
> 2020-04-25T00:41:01.4589519Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-04-25T00:41:01.4590089Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-04-25T00:41:01.4590748Z at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:659)
> 2020-04-25T00:41:01.4591524Z at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
> 2020-04-25T00:41:01.4592062Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1643)
> 2020-04-25T00:41:01.4592597Z at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:35)
> 2020-04-25T00:41:01.4593092Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnce(KafkaProducerTestBase.java:370)
> 2020-04-25T00:41:01.4593680Z at
> org.apache.flink.streaming.connectors.kafka.KafkaProducerTestBase.testExactlyOnceCustomOperator(KafkaProducerTestBase.java:317)
> 2020-04-25T00:41:01.4594450Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-04-25T00:41:01.4595076Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-04-25T00:41:01.4595794Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-04-25T00:41:01.4596622Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-04-25T00:41:01.4597501Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-04-25T00:41:01.4598396Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-04-25T00:41:01.460Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-04-25T00:41:01.4603082Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-04-25T00:41:01.4604023Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-04-25T00:41:01.4604590Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-04-25T00:41:01.4605225Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-04-25T00:41:01.4605902Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-04-25T00:41:01.4606591Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-04-25T00:41:01.4607468Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-04-25T00:41:01.4608577Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-04-25T00:41:01.4609030Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-04-25T00:41:01.4609460Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-04-25T00:41:01.4609842Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-04-25T00:41:01.4610270Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-04-25T00:41:01.4610727Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-04-25T00:41:01.4611147Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-04-25T00:41:01.4611628Z at
>
[jira] [Updated] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-17328: --- Description: JobDetailsHandler * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg * back-pressured for show whether it is back pressured(merge all iths subtasks) was: JobVertexDetailsHandler * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg * back-pressured for show whether it is back pressured(merge all iths subtasks) > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > > JobDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090203#comment-17090203 ] lining edited comment on FLINK-17328 at 4/23/20, 3:40 AM: -- [Gary Yao|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gjy] could you assign it to me? was (Author: lining): [~gary] could you assign it to me? > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > JobVertexDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090203#comment-17090203 ] lining commented on FLINK-17328: [~gary] could you assign it to me? > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > JobVertexDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14815) Expose network metric for sub task in rest api
[ https://issues.apache.org/jira/browse/FLINK-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14815: --- Description: * SubTask ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, inputFloatingBuffersUsage. *** If the subtask is not back pressured, but it is causing backpressure (full input, empty output) *** By comparing exclusive/floating buffers usage, whether all channels are back-pressure or only some of them ** back-pressured for show whether it is back pressured. was: * SubTask ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, inputFloatingBuffersUsage. *** If the subtask is not back pressured, but it is causing backpressure (full input, empty output) *** By comparing exclusive/floating buffers usage, whether all channels are back-pressure or only some of them ** back-pressured for show whether it is back pressured. * Vertex ** pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg ** back-pressured for show whether it is back pressured(merge all iths subtasks) > Expose network metric for sub task in rest api > -- > > Key: FLINK-14815 > URL: https://issues.apache.org/jira/browse/FLINK-14815 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / Network, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > * SubTask > ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, > inputFloatingBuffersUsage. > *** If the subtask is not back pressured, but it is causing backpressure > (full input, empty output) > *** By comparing exclusive/floating buffers usage, whether all channels are > back-pressure or only some of them > ** back-pressured for show whether it is back pressured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-17328: --- Description: JobVertexDetailsHandler * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg * back-pressured for show whether it is back pressured(merge all iths subtasks) was: JobDetailsHandler * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg * back-pressured for show whether it is back pressured(merge all iths subtasks) > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > JobVertexDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14815) Expose network metric for sub task in rest api
[ https://issues.apache.org/jira/browse/FLINK-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14815: --- Summary: Expose network metric for sub task in rest api (was: Expose network metric in rest api) > Expose network metric for sub task in rest api > -- > > Key: FLINK-14815 > URL: https://issues.apache.org/jira/browse/FLINK-14815 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / Network, Runtime / REST >Reporter: lining >Assignee: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > * SubTask > ** pool usage: outPoolUsage, inputExclusiveBuffersUsage, > inputFloatingBuffersUsage. > *** If the subtask is not back pressured, but it is causing backpressure > (full input, empty output) > *** By comparing exclusive/floating buffers usage, whether all channels are > back-pressure or only some of them > ** back-pressured for show whether it is back pressured. > * Vertex > ** pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > ** back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17328) Expose network metric for job vertex in rest api
[ https://issues.apache.org/jira/browse/FLINK-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-17328: --- Parent: FLINK-14712 Issue Type: Sub-task (was: Improvement) > Expose network metric for job vertex in rest api > > > Key: FLINK-17328 > URL: https://issues.apache.org/jira/browse/FLINK-17328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > JobDetailsHandler > * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, > inputFloatingBuffersUsageAvg > * back-pressured for show whether it is back pressured(merge all iths > subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17328) Expose network metric for job vertex in rest api
lining created FLINK-17328: -- Summary: Expose network metric for job vertex in rest api Key: FLINK-17328 URL: https://issues.apache.org/jira/browse/FLINK-17328 Project: Flink Issue Type: Improvement Components: Runtime / Metrics, Runtime / REST Reporter: lining JobDetailsHandler * pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg, inputFloatingBuffersUsageAvg * back-pressured for show whether it is back pressured(merge all iths subtasks) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime In Rest API
[
https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13510:
---
Description:
Now, a user just can see subtask current attempt in the timeline. If job
failover, can not see some has canceled task timeline.
h3. Proposed Changes
* In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
one added field 'attempt'.
* add query parameter show-history, the default value is false. If
show-history is true, information for all attempts including
previous ones will be returned
* URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
* response：
{code:json}
{
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo",
"properties" : {
"id" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"now" : {
"type" : "integer"
},
"subtasks" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo:SubtaskTimeInfo",
"properties" : {
"duration" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"subtask" : {
"type" : "integer"
},
"timestamps" : {
"type" : "object",
"additionalProperties" : {
"type" : "integer"
}
},
"attempt": {
"type" : "integer"
}
}
}
}
}
}
{code}
was:
Now, a user just can see subtask current attempt in the timeline. If job
failover, can not see some has canceled task timeline.
h3. Proposed Changes
* In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
one added field 'attempt-num'.
* add query parameter show-history, the default value is false. If
show-history is true, information for all attempts including
previous ones will be returned
* URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
* response：
{code:json}
{
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo",
"properties" : {
"id" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"now" : {
"type" : "integer"
},
"subtasks" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo:SubtaskTimeInfo",
"properties" : {
"duration" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"subtask" : {
"type" : "integer"
},
"timestamps" : {
"type" : "object",
"additionalProperties" : {
"type" : "integer"
}
},
"attempt-num": {
"type" : "integer"
}
}
}
}
}
}
{code}
> Show fail attempt for subtask in timelime In Rest API
> -
>
> Key: FLINK-13510
> URL: https://issues.apache.org/jira/browse/FLINK-13510
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Now, a user just can see subtask current attempt in the timeline. If job
> failover, can not see some has canceled task timeline.
> h3. Proposed Changes
> * In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
> one added field 'attempt'.
> * add query parameter show-history, the default value is false. If
> show-history is true, information for all attempts including
> previous ones will be returned
> * URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
> * response：
> {code:json}
> {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo",
> "properties" : {
> "id" : {
> "type" : "string"
> },
> "name" : {
> "type" : "string"
> },
> "now" : {
> "type" : "integer"
> },
> "subtasks" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo:SubtaskTimeInfo",
> "properties" : {
> "duration" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "subtask" : {
> "type" : "integer"
> },
> "timestamps" : {
> "type" : "object",
> "additionalProperties" : {
> "type" : "integer"
> }
> },
> "attempt": {
>
[jira] [Updated] (FLINK-14713) Show All Attempts For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* AccessExecutionVertex add method to returns the prior executions
* get prior attempts according to
AccessExecutionVertex.getPriorExecutionAttempts
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskAllExecutionAttemptsDetailsInfo",
"properties" : {
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
},
"taskmanager-id" : {
"type" : "string"
},
"start_time" : {
"type" : "integer"
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* AccessExecutionVertex add method to returns the prior executions
* get prior attempts according to
AccessExecutionVertex.getPriorExecutionAttempts
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
[jira] [Comment Edited] (FLINK-14713) Show All Attempts For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083126#comment-17083126
]
lining edited comment on FLINK-14713 at 4/14/20, 11:35 AM:
---
Hi [~gjy], could you assign it to me and review the PR. Thanks.
was (Author: lining):
[~gjy]
> Show All Attempts For Vertex SubTask In Rest Api
>
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the
> historical attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * AccessExecutionVertex add method to returns the prior executions
> * get prior attempts according to
> AccessExecutionVertex.getPriorExecutionAttempts
> * add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
> * url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
> * response:
> {code:json}
> {
> "attempts" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
> "properties" : {
> "subtask" : {
> "type" : "integer"
> },
> "status" : {
> "type" : "string",
> "enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
> "FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
> },
> "attempt" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "start-time" : {
> "type" : "integer"
> },
> "end-time" : {
> "type" : "integer"
> },
> "duration" : {
> "type" : "integer"
> },
> "metrics" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
> "properties" : {
> "read-bytes" : {
> "type" : "integer"
> },
> "read-bytes-complete" : {
> "type" : "boolean"
> },
> "write-bytes" : {
> "type" : "integer"
> },
> "write-bytes-complete" : {
> "type" : "boolean"
> },
> "read-records" : {
> "type" : "integer"
> },
> "read-records-complete" : {
> "type" : "boolean"
> },
> "write-records" : {
> "type" : "integer"
> },
> "write-records-complete" : {
> "type" : "boolean"
> }
> }
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14713) Show All Attempts For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083126#comment-17083126
]
lining commented on FLINK-14713:
[~gjy]
> Show All Attempts For Vertex SubTask In Rest Api
>
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the
> historical attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * AccessExecutionVertex add method to returns the prior executions
> * get prior attempts according to
> AccessExecutionVertex.getPriorExecutionAttempts
> * add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
> * url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
> * response:
> {code:json}
> {
> "attempts" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
> "properties" : {
> "subtask" : {
> "type" : "integer"
> },
> "status" : {
> "type" : "string",
> "enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
> "FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
> },
> "attempt" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "start-time" : {
> "type" : "integer"
> },
> "end-time" : {
> "type" : "integer"
> },
> "duration" : {
> "type" : "integer"
> },
> "metrics" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
> "properties" : {
> "read-bytes" : {
> "type" : "integer"
> },
> "read-bytes-complete" : {
> "type" : "boolean"
> },
> "write-bytes" : {
> "type" : "integer"
> },
> "write-bytes-complete" : {
> "type" : "boolean"
> },
> "read-records" : {
> "type" : "integer"
> },
> "read-records-complete" : {
> "type" : "boolean"
> },
> "write-records" : {
> "type" : "integer"
> },
> "write-records-complete" : {
> "type" : "boolean"
> }
> }
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime In Rest API
[
https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13510:
---
Description:
Now, a user just can see subtask current attempt in the timeline. If job
failover, can not see some has canceled task timeline.
h3. Proposed Changes
* In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
one added field 'attempt-num'.
* add query parameter show-history, the default value is false. If
show-history is true, information for all attempts including
previous ones will be returned
* URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
* response：
{code:json}
{
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo",
"properties" : {
"id" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"now" : {
"type" : "integer"
},
"subtasks" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo:SubtaskTimeInfo",
"properties" : {
"duration" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"subtask" : {
"type" : "integer"
},
"timestamps" : {
"type" : "object",
"additionalProperties" : {
"type" : "integer"
}
},
"attempt-num": {
"type" : "integer"
}
}
}
}
}
}
{code}
was:
Now, a user just can see subtask current attempt in the timeline. If job
failover, can not see some has canceled task timeline.
h3. Proposed Changes
* In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
one added field 'attempt-num'.
* add query parameter show-history, the default value is false. If
show-history is true, information for all attempts including
previous ones will be returned
* URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
* response：
> Show fail attempt for subtask in timelime In Rest API
> -
>
> Key: FLINK-13510
> URL: https://issues.apache.org/jira/browse/FLINK-13510
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Now, a user just can see subtask current attempt in the timeline. If job
> failover, can not see some has canceled task timeline.
> h3. Proposed Changes
> * In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only
> one added field 'attempt-num'.
> * add query parameter show-history, the default value is false. If
> show-history is true, information for all attempts including
> previous ones will be returned
> * URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true
> * response：
> {code:json}
> {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo",
> "properties" : {
> "id" : {
> "type" : "string"
> },
> "name" : {
> "type" : "string"
> },
> "now" : {
> "type" : "integer"
> },
> "subtasks" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:SubtasksTimesInfo:SubtaskTimeInfo",
> "properties" : {
> "duration" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "subtask" : {
> "type" : "integer"
> },
> "timestamps" : {
> "type" : "object",
> "additionalProperties" : {
> "type" : "integer"
> }
> },
> "attempt-num": {
> "type" : "integer"
> }
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime In Rest API
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-13510: --- Description: Now, a user just can see subtask current attempt in the timeline. If job failover, can not see some has canceled task timeline. h3. Proposed Changes * In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only one added field 'attempt-num'. * add query parameter show-history, the default value is false. If show-history is true, information for all attempts including previous ones will be returned * URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true * response： was: Now, user just can see subtask current attempt in timeline. If job failover, can not see some has cancled task timeline. * In the 'subtasks' array we have objects of type SubtaskTimeInfo with only one added field 'attempt-num'. * add query parameter show-history, default value is false. If show-history is true, information for all attempts including previous ones will be returned * url: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true * response： > Show fail attempt for subtask in timelime In Rest API > - > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Now, a user just can see subtask current attempt in the timeline. If job > failover, can not see some has canceled task timeline. > h3. Proposed Changes > * In the 'subtasks' array, we have objects of type SubtaskTimeInfo with only > one added field 'attempt-num'. > * add query parameter show-history, the default value is false. If > show-history is true, information for all attempts including > previous ones will be returned > * URL: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true > * response： -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime In Rest API
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-13510: --- Description: Now, user just can see subtask current attempt in timeline. If job failover, can not see some has cancled task timeline. * In the 'subtasks' array we have objects of type SubtaskTimeInfo with only one added field 'attempt-num'. * add query parameter show-history, default value is false. If show-history is true, information for all attempts including previous ones will be returned * url: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true * response： was:Now, user just can see subtask current attempt in timeline. If job failover, can not see some has cancled task timeline. > Show fail attempt for subtask in timelime In Rest API > - > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Now, user just can see subtask current attempt in timeline. If job failover, > can not see some has cancled task timeline. > * In the 'subtasks' array we have objects of type SubtaskTimeInfo with only > one added field 'attempt-num'. > * add query parameter show-history, default value is false. If show-history > is true, information for all attempts including > previous ones will be returned > * url: /jobs/:jobid/vertices/:vertexid/subtasktimes?show-history=true > * response： -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show All Attempts For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* AccessExecutionVertex add method to returns the prior executions
* get prior attempts according to
AccessExecutionVertex.getPriorExecutionAttempts
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
[jira] [Updated] (FLINK-14137) Show Attempt List in Vertex SubTask In WebUI
[ https://issues.apache.org/jira/browse/FLINK-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14137: --- Summary: Show Attempt List in Vertex SubTask In WebUI (was: Show Attempt History in Vertex SubTask In WebUI) > Show Attempt List in Vertex SubTask In WebUI > > > Key: FLINK-14137 > URL: https://issues.apache.org/jira/browse/FLINK-14137 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 上午11.32.54.png, 屏幕快照 2019-09-20 > 上午11.32.59.png > > > According to the > [docs|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-vertices-vertexid-subtasks-subtaskindex], > there may exist more than one attempt in a subtask, but there is no way to > get the attempt history list in the REST API, users have no way to know if > the subtask has failed before. > !屏幕快照 2019-09-20 上午11.32.54.png|width=499,height=205! > We can add the Attempt History tab under the Subtasks drawer on the job > vertex page, here is a demo below. > !屏幕快照 2019-09-20 上午11.32.59.png|width=518,height=203! > REST API needed: > add /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts API to > get attempt history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show All Attempts For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Summary: Show All Attempts For Vertex SubTask In Rest Api (was: Show
Historical Attempt For Vertex SubTask In Rest Api)
> Show All Attempts For Vertex SubTask In Rest Api
>
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the
> historical attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i = currentAttemptNum - 1; i >= 0; i--) {
> final AccessExecution currentExecution =
> executionVertex.getPriorExecutionAttempt(i);
> if (currentExecution != null) {
>
> allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
> metricFetcher, jobID, jobVertexID));
> }
> }
> }
> {code}
>
> * add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
> * url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
> * response:
> {code:json}
> {
> "attempts" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
> "properties" : {
> "subtask" : {
> "type" : "integer"
> },
> "status" : {
> "type" : "string",
> "enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
> "FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
> },
> "attempt" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "start-time" : {
> "type" : "integer"
> },
> "end-time" : {
> "type" : "integer"
> },
> "duration" : {
> "type" : "integer"
> },
> "metrics" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
> "properties" : {
> "read-bytes" : {
> "type" : "integer"
> },
> "read-bytes-complete" : {
> "type" : "boolean"
> },
> "write-bytes" : {
> "type" : "integer"
> },
> "write-bytes-complete" : {
> "type" : "boolean"
> },
> "read-records" : {
> "type" : "integer"
> },
> "read-records-complete" : {
> "type" : "boolean"
> },
> "write-records" : {
> "type" : "integer"
> },
> "write-records-complete" : {
> "type" : "boolean"
> }
> }
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show Historical Attempt For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
[jira] [Updated] (FLINK-14713) Show Historical Attempt For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the
historical attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
[jira] [Updated] (FLINK-14713) Show Historical Attempt For Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Summary: Show Historical Attempt For Vertex SubTask In Rest Api (was: Show
Attempt History in Vertex SubTask In Rest Api)
> Show Historical Attempt For Vertex SubTask In Rest Api
> --
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the failed
> attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i = currentAttemptNum - 1; i >= 0; i--) {
> final AccessExecution currentExecution =
> executionVertex.getPriorExecutionAttempt(i);
> if (currentExecution != null) {
>
> allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
> metricFetcher, jobID, jobVertexID));
> }
> }
> }
> {code}
>
> * add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
> * url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
> * response:
> {code:json}
> {
> "attempts" : {
> "type" : "array",
> "items" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
> "properties" : {
> "subtask" : {
> "type" : "integer"
> },
> "status" : {
> "type" : "string",
> "enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
> "FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
> },
> "attempt" : {
> "type" : "integer"
> },
> "host" : {
> "type" : "string"
> },
> "start-time" : {
> "type" : "integer"
> },
> "end-time" : {
> "type" : "integer"
> },
> "duration" : {
> "type" : "integer"
> },
> "metrics" : {
> "type" : "object",
> "id" :
> "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
> "properties" : {
> "read-bytes" : {
> "type" : "integer"
> },
> "read-bytes-complete" : {
> "type" : "boolean"
> },
> "write-bytes" : {
> "type" : "integer"
> },
> "write-bytes-complete" : {
> "type" : "boolean"
> },
> "read-records" : {
> "type" : "integer"
> },
> "read-records-complete" : {
> "type" : "boolean"
> },
> "write-records" : {
> "type" : "integer"
> },
> "write-records-complete" : {
> "type" : "boolean"
> }
> }
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for prior attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
{code:json}
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
{code}
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
```json
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Description:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
* add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* url /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskIndex/attempts
* response:
```json
{
"attempts" : {
"type" : "array",
"items" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:SubtaskExecutionAttemptDetailsInfo",
"properties" : {
"subtask" : {
"type" : "integer"
},
"status" : {
"type" : "string",
"enum" : [ "CREATED", "SCHEDULED", "DEPLOYING", "RUNNING",
"FINISHED", "CANCELING", "CANCELED", "FAILED", "RECONCILING" ]
},
"attempt" : {
"type" : "integer"
},
"host" : {
"type" : "string"
},
"start-time" : {
"type" : "integer"
},
"end-time" : {
"type" : "integer"
},
"duration" : {
"type" : "integer"
},
"metrics" : {
"type" : "object",
"id" :
"urn:jsonschema:org:apache:flink:runtime:rest:messages:job:metrics:IOMetricsInfo",
"properties" : {
"read-bytes" : {
"type" : "integer"
},
"read-bytes-complete" : {
"type" : "boolean"
},
"write-bytes" : {
"type" : "integer"
},
"write-bytes-complete" : {
"type" : "boolean"
},
"read-records" : {
"type" : "integer"
},
"read-records-complete" : {
"type" : "boolean"
},
"write-records" : {
"type" : "integer"
},
"write-records-complete" : {
"type" : "boolean"
}
}
}
}
}
}
}
```
was:
Flink jobs could recovery by failover, but the user couldn't see any
information about the jobs' failure. There isn't information about the failed
attempt.
h3. Proposed Changes
h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
* return subtask all attempt and state
* get prior attempts according to
{code:java}
final AccessExecution execution = executionVertex.getCurrentExecutionAttempt();
final int currentAttemptNum = execution.getAttemptNumber();
if (currentAttemptNum > 0) {
for (int i = currentAttemptNum - 1; i >= 0; i--) {
final AccessExecution currentExecution =
executionVertex.getPriorExecutionAttempt(i);
if (currentExecution != null) {
allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
metricFetcher, jobID, jobVertexID));
}
}
}
{code}
> Show Attempt History in Vertex SubTask In Rest Api
> --
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the failed
> attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i =
[jira] [Updated] (FLINK-14143) Failed Attempts display in the timeline
[ https://issues.apache.org/jira/browse/FLINK-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14143: --- Summary: Failed Attempts display in the timeline (was: Failed Attempt does display in the timeline) > Failed Attempts display in the timeline > --- > > Key: FLINK-14143 > URL: https://issues.apache.org/jira/browse/FLINK-14143 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 下午3.46.40.png > > > There may exist more than one attempt in a subtask, but in the timeline page, > the Web UI can only get and visualize the latest execution attempt of a > subtask timeline, there is no way to get a failed attempt timeline in current > REST API. > !屏幕快照 2019-09-20 下午3.46.40.png|width=453,height=207! > > REST API needed: > add failed attempt time in /jobs/:jobid/vertices/:vertexid/subtasktimes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14143) Failed Attempt does display in the timeline
[ https://issues.apache.org/jira/browse/FLINK-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14143: --- Summary: Failed Attempt does display in the timeline (was: Failed Attempt does not display in the timeline) > Failed Attempt does display in the timeline > --- > > Key: FLINK-14143 > URL: https://issues.apache.org/jira/browse/FLINK-14143 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 下午3.46.40.png > > > There may exist more than one attempt in a subtask, but in the timeline page, > the Web UI can only get and visualize the latest execution attempt of a > subtask timeline, there is no way to get a failed attempt timeline in current > REST API. > !屏幕快照 2019-09-20 下午3.46.40.png|width=453,height=207! > > REST API needed: > add failed attempt time in /jobs/:jobid/vertices/:vertexid/subtasktimes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16863) Sorting descendingly on the last modified date of LogInfo
[ https://issues.apache.org/jira/browse/FLINK-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-16863: --- Summary: Sorting descendingly on the last modified date of LogInfo (was: add lastModified as a field of LogInfo) > Sorting descendingly on the last modified date of LogInfo > - > > Key: FLINK-16863 > URL: https://issues.apache.org/jira/browse/FLINK-16863 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Sorting descendingly on the last modified date could a user be able to see > the most recent files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16863) add lastModified as a field of LogInfo
[ https://issues.apache.org/jira/browse/FLINK-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-16863: --- Parent: FLINK-13987 Issue Type: Sub-task (was: Improvement) > add lastModified as a field of LogInfo > -- > > Key: FLINK-16863 > URL: https://issues.apache.org/jira/browse/FLINK-16863 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Sorting descendingly on the last modified date could a user be able to see > the most recent files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16863) add lastModified as a field of LogInfo
lining created FLINK-16863: -- Summary: add lastModified as a field of LogInfo Key: FLINK-16863 URL: https://issues.apache.org/jira/browse/FLINK-16863 Project: Flink Issue Type: Improvement Components: Runtime / REST Reporter: lining Sorting descendingly on the last modified date could a user be able to see the most recent files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/logs/[filename]
** response: same as taskmanager’s log
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/logs/[filename]
** response: same as jobmanager's log
was:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/log/[filename]
** response: same as jobmanager's log
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/logs/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/logs/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16302) add log list and read log by name for taskmanager
[
https://issues.apache.org/jira/browse/FLINK-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-16302:
---
Description:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/logs/[filename]
** response: same as taskmanager’s log
was:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
> add log list and read log by name for taskmanager
> -
>
> Key: FLINK-16302
> URL: https://issues.apache.org/jira/browse/FLINK-16302
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Assignee: lining
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/logs/[filename]
> ** response: same as taskmanager’s log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-11546) Add option to manually set job ID in CLI
[ https://issues.apache.org/jira/browse/FLINK-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048772#comment-17048772 ] lining commented on FLINK-11546: ping [~trohrmann] > Add option to manually set job ID in CLI > > > Key: FLINK-11546 > URL: https://issues.apache.org/jira/browse/FLINK-11546 > Project: Flink > Issue Type: Sub-task > Components: Command Line Client >Affects Versions: 1.7.0 >Reporter: Ufuk Celebi >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add an option to specify the job ID during job submissions via the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048771#comment-17048771
]
lining edited comment on FLINK-13987 at 3/2/20 4:05 AM:
ping [~gjy]
was (Author: lining):
ping [~gary].
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048771#comment-17048771
]
lining commented on FLINK-13987:
ping [~gary].
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15314) To refactor duplicated code in TaskManagerDetailsHandler#createTaskManagerMetricsInfo
[ https://issues.apache.org/jira/browse/FLINK-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048768#comment-17048768 ] lining commented on FLINK-15314: Hi [~gjy] , I encountered a specific problem: If some developers make changes to the metrics in the place where these metrics are defined(ps: in MetricUtils). But it forgot to update code in the link; the rest API will get the wrong information. If we define these keys in MetricUtils, rest API call MetricUtils.xxx to get the key. Then we only need to maintain them in one place. > To refactor duplicated code in > TaskManagerDetailsHandler#createTaskManagerMetricsInfo > - > > Key: FLINK-15314 > URL: https://issues.apache.org/jira/browse/FLINK-15314 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > As > [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] > show, use hard codes to define metrics' name. Could we unify where to > register and get metrics? > * propose > ** move metrics'name define in MetricUtils to MetricNames > ** the all get the name form MetricNames -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15314) To refactor duplicated code in TaskManagerDetailsHandler#createTaskManagerMetricsInfo
[ https://issues.apache.org/jira/browse/FLINK-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047619#comment-17047619 ] lining commented on FLINK-15314: Thanks for your reply, I have updated the code link. > To refactor duplicated code in > TaskManagerDetailsHandler#createTaskManagerMetricsInfo > - > > Key: FLINK-15314 > URL: https://issues.apache.org/jira/browse/FLINK-15314 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > As > [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] > show, use hard codes to define metrics' name. Could we unify where to > register and get metrics? > * propose > ** move metrics'name define in MetricUtils to MetricNames > ** the all get the name form MetricNames -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15314) To refactor duplicated code in TaskManagerDetailsHandler#createTaskManagerMetricsInfo
[ https://issues.apache.org/jira/browse/FLINK-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15314: --- Description: As [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] show, use hard codes to define metrics' name. Could we unify where to register and get metrics? * propose ** move metrics'name define in MetricUtils to MetricNames ** the all get the name form MetricNames was: As [code|[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] show, use hard codes to define metrics' name. Could we unify where to register and get metrics? * propose ** move metrics'name define in MetricUtils to MetricNames ** the all get the name form MetricNames > To refactor duplicated code in > TaskManagerDetailsHandler#createTaskManagerMetricsInfo > - > > Key: FLINK-15314 > URL: https://issues.apache.org/jira/browse/FLINK-15314 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > As > [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] > show, use hard codes to define metrics' name. Could we unify where to > register and get metrics? > * propose > ** move metrics'name define in MetricUtils to MetricNames > ** the all get the name form MetricNames -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15314) To refactor duplicated code in TaskManagerDetailsHandler#createTaskManagerMetricsInfo
[ https://issues.apache.org/jira/browse/FLINK-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15314: --- Description: As [code|[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] show, use hard codes to define metrics' name. Could we unify where to register and get metrics? * propose ** move metrics'name define in MetricUtils to MetricNames ** the all get the name form MetricNames was: As [code|[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L123-L140]] show, use hard codes to define metrics' name. Could we unify where to register and get metrics? * propose ** move metrics'name define in MetricUtils to MetricNames ** the all get the name form MetricNames > To refactor duplicated code in > TaskManagerDetailsHandler#createTaskManagerMetricsInfo > - > > Key: FLINK-15314 > URL: https://issues.apache.org/jira/browse/FLINK-15314 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / REST >Reporter: lining >Priority: Major > > As > [code|[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerDetailsHandler.java#L119-L142] > show, use hard codes to define metrics' name. Could we unify where to > register and get metrics? > * propose > ** move metrics'name define in MetricUtils to MetricNames > ** the all get the name form MetricNames -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add log list and read log by name
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Summary: add log list and read log by name (was: add new logs api, see
more log files)
> add log list and read log by name
> -
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16303) add log list and read log by name for jobmanager
[
https://issues.apache.org/jira/browse/FLINK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-16303:
---
Description:
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/log/[filename]
** response: same as jobmanager's log
> add log list and read log by name for jobmanager
>
>
> Key: FLINK-16303
> URL: https://issues.apache.org/jira/browse/FLINK-16303
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16302) add log list and read log by name for taskmanager
[
https://issues.apache.org/jira/browse/FLINK-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-16302:
---
Description:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
> add log list and read log by name for taskmanager
> -
>
> Key: FLINK-16302
> URL: https://issues.apache.org/jira/browse/FLINK-16302
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16303) add log list and read log by name for jobmanager
lining created FLINK-16303: -- Summary: add log list and read log by name for jobmanager Key: FLINK-16303 URL: https://issues.apache.org/jira/browse/FLINK-16303 Project: Flink Issue Type: Sub-task Components: Runtime / REST Reporter: lining -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16302) add log list and read log by name for taskmanager
lining created FLINK-16302: -- Summary: add log list and read log by name for taskmanager Key: FLINK-16302 URL: https://issues.apache.org/jira/browse/FLINK-16302 Project: Flink Issue Type: Sub-task Components: Runtime / REST Reporter: lining -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Summary: add new logs api, see more log files (was: add new logs api, see
more log files and can see logs by pages )
> add new logs api, see more log files
>
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/log/[filename]
** response: same as jobmanager's log
was:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/log/[filename]
** response: same as jobmanager's log
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* read taskmanager log file
** /taskmanagers/log/[filename]
** response: same as taskmanager’s log
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
* read jobmanager log file
** /jobmanager/log/[filename]
** response: same as jobmanager's log
was:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * read taskmanager log file
> ** /taskmanagers/log/[filename]
> ** response: same as taskmanager’s log
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
> * read jobmanager log file
> ** /jobmanager/log/[filename]
> ** response: same as jobmanager's log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
Current log API returns all contents，it will block or not work when the file is
large. It's unfriendly for the user.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there aren't this contents.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
was:
As the job running, the log files are becoming large.
Current log API returns all contents，it will block or not work when the file is
large. It's unfriendly for the user.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there aren't this contents.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* see taskmanager log file by range
** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
**
{code:java}
{
"data": "logcontent",
"file_size": 342882
}
{code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}
{code}
* see jobmanager log file by range
** /jobmanager/logs/:filename?start=[start]=[count]
**
{code:java}
{
"data": "logcontent",
"file_size": 342882
}
{code}
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log API returns all contents，it will block or not work when the file
> is large. It's unfriendly for the user.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there aren't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
was:
As the job running, the log files are becoming large.
Current log API returns all contents，it will block or not work when the file is
large. It's unfriendly for the user.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there aren't this contents.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there aren't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-13987:
---
Description:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there isn't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
was:
As the job running, the log files are becoming large.
As the application runs on JVM, sometimes the user needs to see the log of GC,
but there aren't this content.
Above all, we need new apis:
* list taskmanager all log file
** /taskmanagers/taskmanagerid/logs
**
{code:java}
{
"logs": [
{
"name": "taskmanager.log",
"size": 12529
}
]
} {code}
* list jobmanager all log file
** /jobmanager/logs
**
{code:java}
{
"logs": [
{
"name": "jobmanager.log",
"size": 12529
}
]
}{code}
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there isn't this content.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13987) add new logs api, see more log files and can see logs by pages
[
https://issues.apache.org/jira/browse/FLINK-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044241#comment-17044241
]
lining commented on FLINK-13987:
As FLIP-103 has approved, could someone assign it to me?
> add new logs api, see more log files and can see logs by pages
> ---
>
> Key: FLINK-13987
> URL: https://issues.apache.org/jira/browse/FLINK-13987
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> As the job running, the log files are becoming large.
> Current log API returns all contents，it will block or not work when the file
> is large. It's unfriendly for the user.
> As the application runs on JVM, sometimes the user needs to see the log of
> GC, but there aren't this contents.
> Above all, we need new apis:
> * list taskmanager all log file
> ** /taskmanagers/taskmanagerid/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "taskmanager.log",
> "size": 12529
> }
> ]
> } {code}
> * see taskmanager log file by range
> ** /taskmanagers/taskmanagerid/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
> * list jobmanager all log file
> ** /jobmanager/logs
> **
> {code:java}
> {
> "logs": [
> {
> "name": "jobmanager.log",
> "size": 12529
> }
> ]
> }
> {code}
> * see jobmanager log file by range
> ** /jobmanager/logs/:filename?start=[start]=[count]
> **
> {code:java}
> {
> "data": "logcontent",
> "file_size": 342882
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-11546) Add option to manually set job ID in CLI
[ https://issues.apache.org/jira/browse/FLINK-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044160#comment-17044160 ] lining commented on FLINK-11546: [~uce] could you assign it to me? > Add option to manually set job ID in CLI > > > Key: FLINK-11546 > URL: https://issues.apache.org/jira/browse/FLINK-11546 > Project: Flink > Issue Type: Sub-task > Components: Command Line Client >Affects Versions: 1.7.0 >Reporter: Ufuk Celebi >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add an option to specify the job ID during job submissions via the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16184) Submit job with fixed/configurable JobID
[ https://issues.apache.org/jira/browse/FLINK-16184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining closed FLINK-16184. -- Resolution: Duplicate > Submit job with fixed/configurable JobID > - > > Key: FLINK-16184 > URL: https://issues.apache.org/jira/browse/FLINK-16184 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: lining >Priority: Major > > Now the user could define jobId when the user submit job by JarRunHandler. > But we couldn't define jobId when submit job by flink run. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-11546) Add option to manually set job ID in CLI
[ https://issues.apache.org/jira/browse/FLINK-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040779#comment-17040779 ] lining commented on FLINK-11546: [~yanghua], how is this going now? > Add option to manually set job ID in CLI > > > Key: FLINK-11546 > URL: https://issues.apache.org/jira/browse/FLINK-11546 > Project: Flink > Issue Type: Sub-task > Components: Command Line Client >Affects Versions: 1.7.0 >Reporter: Ufuk Celebi >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add an option to specify the job ID during job submissions via the CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16184) Submit job with fixed/configurable JobID
[ https://issues.apache.org/jira/browse/FLINK-16184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-16184: --- Description: Now the user could define jobId when the user submit job by JarRunHandler. But we couldn't define jobId when submit job by flink run. > Submit job with fixed/configurable JobID > - > > Key: FLINK-16184 > URL: https://issues.apache.org/jira/browse/FLINK-16184 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: lining >Priority: Major > > Now the user could define jobId when the user submit job by JarRunHandler. > But we couldn't define jobId when submit job by flink run. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16184) Submit job with fixed/configurable JobID
lining created FLINK-16184: -- Summary: Submit job with fixed/configurable JobID Key: FLINK-16184 URL: https://issues.apache.org/jira/browse/FLINK-16184 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: lining -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14138) Show Pending Slots in Job Detail
[ https://issues.apache.org/jira/browse/FLINK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14138: --- Component/s: Runtime / REST > Show Pending Slots in Job Detail > > > Key: FLINK-14138 > URL: https://issues.apache.org/jira/browse/FLINK-14138 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 下午12.04.00.png, 屏幕快照 2019-09-20 > 下午12.04.05.png > > > It is hard to troubleshoot when all subtasks are always on the SCHEDULED > status(just like the screenshot below) when users submit a job. > !屏幕快照 2019-09-20 下午12.04.00.png|width=494,height=258! > The most common reason for this problem is that vertex has applied for more > resources than the cluster has. A pending slots tab could help users to check > which vertex or subtask is blocked. > !屏幕快照 2019-09-20 下午12.04.05.png|width=576,height=163! > > REST API needed: > add /jobs/:jobid/pending-slots API to get pending slots data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14730) Add pending slots for job
[ https://issues.apache.org/jira/browse/FLINK-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14730: --- Parent: FLINK-14138 Issue Type: Sub-task (was: Improvement) > Add pending slots for job > - > > Key: FLINK-14730 > URL: https://issues.apache.org/jira/browse/FLINK-14730 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > *Current* > If the resource requested by the job can‘t be satisfied by the cluster, the > job will remain in the scheduling state. > The user couldn't know the scheduler is blocked by which slot request. > *Proposal* > We could add a rest handler to show information about pending requests in > SlotPoolImpl. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14730) Add pending slots for job
[ https://issues.apache.org/jira/browse/FLINK-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14730: --- Component/s: (was: Runtime / Web Frontend) > Add pending slots for job > - > > Key: FLINK-14730 > URL: https://issues.apache.org/jira/browse/FLINK-14730 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: lining >Priority: Major > > *Current* > If the resource requested by the job can‘t be satisfied by the cluster, the > job will remain in the scheduling state. > The user couldn't know the scheduler is blocked by which slot request. > *Proposal* > We could add a rest handler to show information about pending requests in > SlotPoolImpl. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-13510: --- Component/s: (was: Runtime / Web Frontend) > Show fail attempt for subtask in timelime > - > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Now, user just can see subtask current attempt in timeline. If job failover, > can not see some has cancled task timeline. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime In Rest API
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-13510: --- Summary: Show fail attempt for subtask in timelime In Rest API (was: Show fail attempt for subtask in timelime) > Show fail attempt for subtask in timelime In Rest API > - > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: lining >Priority: Major > > Now, user just can see subtask current attempt in timeline. If job failover, > can not see some has cancled task timeline. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14137) Show Attempt History in Vertex SubTask In WebUI
[ https://issues.apache.org/jira/browse/FLINK-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14137: --- Summary: Show Attempt History in Vertex SubTask In WebUI (was: Show Attempt History in Vertex SubTask) > Show Attempt History in Vertex SubTask In WebUI > --- > > Key: FLINK-14137 > URL: https://issues.apache.org/jira/browse/FLINK-14137 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 上午11.32.54.png, 屏幕快照 2019-09-20 > 上午11.32.59.png > > > According to the > [docs|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-vertices-vertexid-subtasks-subtaskindex], > there may exist more than one attempt in a subtask, but there is no way to > get the attempt history list in the REST API, users have no way to know if > the subtask has failed before. > !屏幕快照 2019-09-20 上午11.32.54.png|width=499,height=205! > We can add the Attempt History tab under the Subtasks drawer on the job > vertex page, here is a demo below. > !屏幕快照 2019-09-20 上午11.32.59.png|width=518,height=203! > REST API needed: > add /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts API to > get attempt history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Component/s: (was: Runtime / Web Frontend)
> Show Attempt History in Vertex SubTask In Rest Api
> --
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the failed
> attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i = currentAttemptNum - 1; i >= 0; i--) {
> final AccessExecution currentExecution =
> executionVertex.getPriorExecutionAttempt(i);
> if (currentExecution != null) {
>
> allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
> metricFetcher, jobID, jobVertexID));
> }
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask In Rest Api
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Summary: Show Attempt History in Vertex SubTask In Rest Api (was: Show
Attempt History in Vertex SubTask)
> Show Attempt History in Vertex SubTask In Rest Api
> --
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST, Runtime / Web Frontend
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the failed
> attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i = currentAttemptNum - 1; i >= 0; i--) {
> final AccessExecution currentExecution =
> executionVertex.getPriorExecutionAttempt(i);
> if (currentExecution != null) {
>
> allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
> metricFetcher, jobID, jobVertexID));
> }
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13510) Show fail attempt for subtask in timelime
[ https://issues.apache.org/jira/browse/FLINK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-13510: --- Parent: FLINK-16050 Issue Type: Sub-task (was: Improvement) > Show fail attempt for subtask in timelime > - > > Key: FLINK-13510 > URL: https://issues.apache.org/jira/browse/FLINK-13510 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Runtime / Web Frontend >Reporter: lining >Priority: Major > > Now, user just can see subtask current attempt in timeline. If job failover, > can not see some has cancled task timeline. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14143) Failed Attempt does not display in the timeline
[ https://issues.apache.org/jira/browse/FLINK-14143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14143: --- Parent: FLINK-16050 Issue Type: Sub-task (was: Improvement) > Failed Attempt does not display in the timeline > --- > > Key: FLINK-14143 > URL: https://issues.apache.org/jira/browse/FLINK-14143 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 下午3.46.40.png > > > There may exist more than one attempt in a subtask, but in the timeline page, > the Web UI can only get and visualize the latest execution attempt of a > subtask timeline, there is no way to get a failed attempt timeline in current > REST API. > !屏幕快照 2019-09-20 下午3.46.40.png|width=453,height=207! > > REST API needed: > add failed attempt time in /jobs/:jobid/vertices/:vertexid/subtasktimes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14713) Show Attempt History in Vertex SubTask
[
https://issues.apache.org/jira/browse/FLINK-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14713:
---
Parent: FLINK-16050
Issue Type: Sub-task (was: Improvement)
> Show Attempt History in Vertex SubTask
> --
>
> Key: FLINK-14713
> URL: https://issues.apache.org/jira/browse/FLINK-14713
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST, Runtime / Web Frontend
>Reporter: lining
>Priority: Major
>
> Flink jobs could recovery by failover, but the user couldn't see any
> information about the jobs' failure. There isn't information about the failed
> attempt.
> h3. Proposed Changes
> h4. Add SubtaskAllExecutionAttemptsDetailsHandler for failed attempt
> * return subtask all attempt and state
> * get prior attempts according to
> {code:java}
> final AccessExecution execution =
> executionVertex.getCurrentExecutionAttempt();
> final int currentAttemptNum = execution.getAttemptNumber();
> if (currentAttemptNum > 0) {
> for (int i = currentAttemptNum - 1; i >= 0; i--) {
> final AccessExecution currentExecution =
> executionVertex.getPriorExecutionAttempt(i);
> if (currentExecution != null) {
>
> allAttempts.add(SubtaskExecutionAttemptDetailsInfo.create(currentExecution,
> metricFetcher, jobID, jobVertexID));
> }
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14137) Show Attempt History in Vertex SubTask
[ https://issues.apache.org/jira/browse/FLINK-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14137: --- Parent: (was: FLINK-14713) Issue Type: Task (was: Sub-task) > Show Attempt History in Vertex SubTask > -- > > Key: FLINK-14137 > URL: https://issues.apache.org/jira/browse/FLINK-14137 > Project: Flink > Issue Type: Task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 上午11.32.54.png, 屏幕快照 2019-09-20 > 上午11.32.59.png > > > According to the > [docs|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-vertices-vertexid-subtasks-subtaskindex], > there may exist more than one attempt in a subtask, but there is no way to > get the attempt history list in the REST API, users have no way to know if > the subtask has failed before. > !屏幕快照 2019-09-20 上午11.32.54.png|width=499,height=205! > We can add the Attempt History tab under the Subtasks drawer on the job > vertex page, here is a demo below. > !屏幕快照 2019-09-20 上午11.32.59.png|width=518,height=203! > REST API needed: > add /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts API to > get attempt history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14137) Show Attempt History in Vertex SubTask
[ https://issues.apache.org/jira/browse/FLINK-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14137: --- Parent: FLINK-16050 Issue Type: Sub-task (was: Task) > Show Attempt History in Vertex SubTask > -- > > Key: FLINK-14137 > URL: https://issues.apache.org/jira/browse/FLINK-14137 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Yadong Xie >Priority: Major > Attachments: 屏幕快照 2019-09-20 上午11.32.54.png, 屏幕快照 2019-09-20 > 上午11.32.59.png > > > According to the > [docs|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-vertices-vertexid-subtasks-subtaskindex], > there may exist more than one attempt in a subtask, but there is no way to > get the attempt history list in the REST API, users have no way to know if > the subtask has failed before. > !屏幕快照 2019-09-20 上午11.32.54.png|width=499,height=205! > We can add the Attempt History tab under the Subtasks drawer on the job > vertex page, here is a demo below. > !屏幕快照 2019-09-20 上午11.32.59.png|width=518,height=203! > REST API needed: > add /jobs/:jobid/vertices/:vertexid/subtasks/:subtaskindex/attempts API to > get attempt history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16050) Add Attempt Information
lining created FLINK-16050: -- Summary: Add Attempt Information Key: FLINK-16050 URL: https://issues.apache.org/jira/browse/FLINK-16050 Project: Flink Issue Type: Improvement Components: Runtime / REST, Runtime / Web Frontend Reporter: lining According to the [docs|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-vertices-vertexid-subtasks-subtaskindex], there may exist more than one attempt in a subtask, but there is no way to get the attempt history list in the REST API, users have no way to know if the subtask has failed before. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14127) Better BackPressure Detection in WebUI
[ https://issues.apache.org/jira/browse/FLINK-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14127: --- Parent: FLINK-14712 Issue Type: Sub-task (was: Improvement) > Better BackPressure Detection in WebUI > -- > > Key: FLINK-14127 > URL: https://issues.apache.org/jira/browse/FLINK-14127 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Yadong Xie >Priority: Major > Fix For: 1.11.0 > > Attachments: 屏幕快照 2019-09-19 下午6.00.05.png, 屏幕快照 2019-09-19 > 下午6.00.57.png, 屏幕快照 2019-09-19 下午6.01.43.png > > > According to the > [Document|https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/back_pressure.html], > the backpressure monitor only triggered on request and it is currently not > available via metrics. This means that in the web UI we have no way to show > all the backpressure state of all vertexes at the same time. The users need > to click every vertex to get its backpressure state. > !屏幕快照 2019-09-19 下午6.00.05.png|width=510,height=197! > In Flink 1.9.0 and above, there are four metrics available(outPoolUsage, > inPoolUsage, floatingBuffersUsage, exclusiveBuffersUsage), we can use these > metrics to determine if there are possible backpressure, and then use the > backpressure REST API to confirm it. > Here is a table get from > [https://flink.apache.org/2019/07/23/flink-network-stack-2.html] > !屏幕快照 2019-09-19 下午6.00.57.png|width=516,height=304! > > We can display the possible backpressure status on the vertex graph, thus > users can get all the vertex backpressure states and locate the potential > problem quickly. > > !屏幕快照 2019-09-19 下午6.01.43.png|width=572,height=277! > > REST API needed: > add outPoolUsage, inPoolUsage, floatingBuffersUsage, exclusiveBuffersUsage > metrics for each vertex in the /jobs/:jobId API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15422) Expose Jvm Metaspace、Compressed Class Space、Eden Space、Survivor Space、Old Gen metric
[ https://issues.apache.org/jira/browse/FLINK-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028790#comment-17028790 ] lining commented on FLINK-15422: We could get these information from _ManagementFactory.getMemoryPoolMXBeans()._ > Expose Jvm Metaspace、Compressed Class Space、Eden Space、Survivor Space、Old Gen > metric > > > Key: FLINK-15422 > URL: https://issues.apache.org/jira/browse/FLINK-15422 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: lining >Priority: Major > > Current, we just expose JVM the metrics of heap and non-heap. But users > couldn't get details of JVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15752) Backpressure stats sometimes broken in WebUI
[ https://issues.apache.org/jira/browse/FLINK-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028771#comment-17028771 ] lining commented on FLINK-15752: [~NicoK], could you show the result from rest API? > Backpressure stats sometimes broken in WebUI > > > Key: FLINK-15752 > URL: https://issues.apache.org/jira/browse/FLINK-15752 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Nico Kruber >Priority: Major > Attachments: backpressure-stats.png > > > The backpressure monitor shows two values: ratio and status. It looks like > they are not always in sync. See below (for low ratios): > !backpressure-stats.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-15752) Backpressure stats sometimes broken in WebUI
[ https://issues.apache.org/jira/browse/FLINK-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15752: --- Comment: was deleted (was: [~NicoK] could you show the result from the rest API?) > Backpressure stats sometimes broken in WebUI > > > Key: FLINK-15752 > URL: https://issues.apache.org/jira/browse/FLINK-15752 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Nico Kruber >Priority: Major > Attachments: backpressure-stats.png > > > The backpressure monitor shows two values: ratio and status. It looks like > they are not always in sync. See below (for low ratios): > !backpressure-stats.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15752) Backpressure stats sometimes broken in WebUI
[ https://issues.apache.org/jira/browse/FLINK-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028760#comment-17028760 ] lining commented on FLINK-15752: [~NicoK] could you show the result from the rest API? > Backpressure stats sometimes broken in WebUI > > > Key: FLINK-15752 > URL: https://issues.apache.org/jira/browse/FLINK-15752 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.10.0 >Reporter: Nico Kruber >Priority: Major > Attachments: backpressure-stats.png > > > The backpressure monitor shows two values: ratio and status. It looks like > they are not always in sync. See below (for low ratios): > !backpressure-stats.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15628) Fix initialize webSubmissionHandlers list in WebSubmissionExtension with correct size
[ https://issues.apache.org/jira/browse/FLINK-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019192#comment-17019192 ] lining commented on FLINK-15628: Hi, [~GJL] could you assign it to me? > Fix initialize webSubmissionHandlers list in WebSubmissionExtension with > correct size > - > > Key: FLINK-15628 > URL: https://issues.apache.org/jira/browse/FLINK-15628 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [code|[https://github.com/apache/flink/blob/master/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebSubmissionExtension.java#L64]] > here needs to be 6. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15423) Show the detail of submitting job fail
[
https://issues.apache.org/jira/browse/FLINK-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-15423:
---
Description:
Now when the user fails to submit the job, could we return the details of the
failure?
For example:
- in Flink web UI
- url:
[http://localhost:8081/jars/ac2873e5-1f5d-493b-97b3-2c658261ee68_TopSpeedWindowing.jar/run?entry-class=org.apache.flink.streaming.examples.windowing.TopSpeedWindowing=214748364]
- result: \{"errors":["Internal server error.","(JobManagerRunnerImpl.java:152)\n\tat
org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84)\n\tat
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:379)\n\tat
org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)\n\t...
7 more\nCaused by: java.lang.IllegalArgumentException: Operator parallelism
not within bounds: 214748364\n\tat
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)\n\tat
org.apache.flink.runtime.state.KeyGroupRangeAssignment.checkParallelismPreconditions(KeyGroupRangeAssignment.java:141)\n\tat
org.apache.flink.runtime.state.KeyGroupRangeAssignment.computeDefaultMaxParallelism(KeyGroupRangeAssignment.java:131)\n\tat
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:184)\n\tat
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:806)\n\tat
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:228)\n\tat
org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:253)\n\tat
org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:225)\n\tat
org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:213)\n\tat
org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:117)\n\tat
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:105)\n\tat
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)\n\tat
org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:266)\n\tat
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)\n\tat
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)\n\tat
org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.(JobManagerRunnerImpl.java:146)\n\t...
10 more\n\nEnd of exception on server side>"]}
was:Now when the user fails to submit the job, it just returns: unable to
load the requested file. Could we return the details of the failure?
> Show the detail of submitting job fail
> --
>
> Key: FLINK-15423
> URL: https://issues.apache.org/jira/browse/FLINK-15423
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: lining
>Priority: Major
>
> Now when the user fails to submit the job, could we return the details of the
> failure?
> For example:
> - in Flink web UI
> - url:
> [http://localhost:8081/jars/ac2873e5-1f5d-493b-97b3-2c658261ee68_TopSpeedWindowing.jar/run?entry-class=org.apache.flink.streaming.examples.windowing.TopSpeedWindowing=214748364]
> - result: \{"errors":["Internal server error."," side:\norg.apache.flink.runtime.client.JobSubmissionException: Failed to
> submit job.\n\tat
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$internalSubmitJob$3(Dispatcher.java:336)\n\tat
>
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)\n\tat
>
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)\n\tat
>
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)\n\tat
> akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)\n\tat
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)\n\tat
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n\tat
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n\tat
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\nCaused
> by: java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobExecutionException: Could not set up
> JobManager\n\tat
> org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36)\n\tat
>
>
[jira] [Updated] (FLINK-15315) Add test case for rest
[ https://issues.apache.org/jira/browse/FLINK-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15315: --- Description: 1. Handler which has no test: * ClusterConfigHandler * ClusterOverviewHandler * DashboardConfigHandler * ShutdownHandler * CheckpointConfigHandler * CheckpointingStatisticsHandler * CheckpointStatisticDetailsHandler * TaskCheckpointStatisticDetailsHandler * RescalingHandlers * JobAccumulatorsHandler * JobDetailsHandler * JobIdsHandler * JobPlanHandler * JobsOverviewHandler * JobVertexAccumulatorsHandler * JobVertexDetailsHandler * JobVertexTaskManagersHandler * SubtasksAllAccumulatorsHandler * SubtasksTimesHandler * TaskManagerDetailsHandler * TaskManagerLogFileHandler * TaskManagersHandler * TaskManagerStdoutFileHandler 2. Some rest server's handlers' data comes from runtime metrics. Currently, if the runtime metrics have updated, these handlers will not be aware of these changes. How can we limit this update error by the test? Such as [input group and output group of the task metric are reversed|https://issues.apache.org/jira/browse/FLINK-15063]. > Add test case for rest > -- > > Key: FLINK-15315 > URL: https://issues.apache.org/jira/browse/FLINK-15315 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Tests >Reporter: lining >Priority: Major > > 1. Handler which has no test: > * ClusterConfigHandler > * ClusterOverviewHandler > * DashboardConfigHandler > * ShutdownHandler > * CheckpointConfigHandler > * CheckpointingStatisticsHandler > * CheckpointStatisticDetailsHandler > * TaskCheckpointStatisticDetailsHandler > * RescalingHandlers > * JobAccumulatorsHandler > * JobDetailsHandler > * JobIdsHandler > * JobPlanHandler > * JobsOverviewHandler > * JobVertexAccumulatorsHandler > * JobVertexDetailsHandler > * JobVertexTaskManagersHandler > * SubtasksAllAccumulatorsHandler > * SubtasksTimesHandler > * TaskManagerDetailsHandler > * TaskManagerLogFileHandler > * TaskManagersHandler > * TaskManagerStdoutFileHandler > 2. Some rest server's handlers' data comes from runtime metrics. Currently, > if the runtime metrics have updated, these handlers will not be aware of > these changes. How can we limit this update error by the test? Such as [input > group and output group of the task metric are > reversed|https://issues.apache.org/jira/browse/FLINK-15063]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15628) Fix initialize webSubmissionHandlers list in WebSubmissionExtension with correct size
[ https://issues.apache.org/jira/browse/FLINK-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017823#comment-17017823 ] lining commented on FLINK-15628: cc [~gjy] > Fix initialize webSubmissionHandlers list in WebSubmissionExtension with > correct size > - > > Key: FLINK-15628 > URL: https://issues.apache.org/jira/browse/FLINK-15628 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: lining >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [code|[https://github.com/apache/flink/blob/master/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebSubmissionExtension.java#L64]] > here needs to be 6. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15628) Fix initialize webSubmissionHandlers list in WebSubmissionExtension with correct size
lining created FLINK-15628: -- Summary: Fix initialize webSubmissionHandlers list in WebSubmissionExtension with correct size Key: FLINK-15628 URL: https://issues.apache.org/jira/browse/FLINK-15628 Project: Flink Issue Type: Improvement Components: Runtime / REST Reporter: lining [code|[https://github.com/apache/flink/blob/master/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebSubmissionExtension.java#L64]] here needs to be 6. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-15489) Cannot update jobmanager/taskmanager logs
[ https://issues.apache.org/jira/browse/FLINK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15489: --- Comment: was deleted (was: As [~vthinkxie] said, it's to reduce the pressure of the rest server. As the API for refresh log is the same as the API for getting log. So to create a new API just for refresh is better.) > Cannot update jobmanager/taskmanager logs > - > > Key: FLINK-15489 > URL: https://issues.apache.org/jira/browse/FLINK-15489 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.9.1, 1.10.0 >Reporter: Dawid Wysakowicz >Priority: Blocker > Fix For: 1.10.0 > > Attachments: screenshot-1.png > > > There is no way to query the latest state of logs of jobmanager/taskmanager. > The Web UI show only the first version that was ever displayed. > How to reproduce: > * (not sure if necessary) configure logback as described here: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster > * start a cluster > * show jobmanager logs in the Web UI > * run example job > * check again the jobmanager logs, there is no trace of the job. Clicking the > refresh button does not help -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15489) Cannot update jobmanager/taskmanager logs
[ https://issues.apache.org/jira/browse/FLINK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011667#comment-17011667 ] lining commented on FLINK-15489: As [~vthinkxie] said, it's to reduce the pressure of the rest server. As the API for refresh log is the same as the API for getting log. So to create a new API just for refresh is better. > Cannot update jobmanager/taskmanager logs > - > > Key: FLINK-15489 > URL: https://issues.apache.org/jira/browse/FLINK-15489 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.9.1, 1.10.0 >Reporter: Dawid Wysakowicz >Priority: Blocker > Fix For: 1.10.0 > > Attachments: screenshot-1.png > > > There is no way to query the latest state of logs of jobmanager/taskmanager. > The Web UI show only the first version that was ever displayed. > How to reproduce: > * (not sure if necessary) configure logback as described here: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster > * start a cluster > * show jobmanager logs in the Web UI > * run example job > * check again the jobmanager logs, there is no trace of the job. Clicking the > refresh button does not help -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15170) WebFrontendITCase.testCancelYarn fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005207#comment-17005207
]
lining edited comment on FLINK-15170 at 12/30/19 9:17 AM:
--
Discover testCancelYarn and testCancel all fail.
Method testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
Methoud testCancelYarn failed because java.util.concurrent.TimeoutException:
Connection failed is located at WebFrontendITCase.java:324.
Possible cause: If maven test first runs testCancelYarn, its request to cancel
fail, then there's one job is running, so testCancel asserts no running job
fail.
was (Author: lining):
Discover testCancelYarn and testCancel all fail.
Method testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
Methoud testCancelYarn failed because java.util.concurrent.TimeoutException:
Connection failed is located at WebFrontendITCase.java:324.
If maven test first runs testCancelYarn, its request to cancel fail, then
there's one job is running, so testCancel asserts no running job fail.
> WebFrontendITCase.testCancelYarn fails on travis
>
>
> Key: FLINK-15170
> URL: https://issues.apache.org/jira/browse/FLINK-15170
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Web Frontend
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: lining
>Priority: Major
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/622756846/log.txt
> {code}
> 23:59:33.299 [INFO] Running
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:33.365 [INFO] Running
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:35.379 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 2.012 s - in
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:38.802 [ERROR] Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time
> elapsed: 5.5 s <<< FAILURE! - in
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:38.802 [ERROR]
> testCancelYarn(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.436 s <<< ERROR!
> java.util.concurrent.TimeoutException: Connection failed
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancelYarn(WebFrontendITCase.java:324)
> 23:59:38.802 [ERROR]
> testCancel(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.003 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancel(WebFrontendITCase.java:240)
> 23:59:39.193 [INFO]
> 23:59:39.193 [INFO] Results:
> 23:59:39.193 [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15170) WebFrontendITCase.testCancelYarn fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005207#comment-17005207
]
lining edited comment on FLINK-15170 at 12/30/19 9:13 AM:
--
Discover testCancelYarn and testCancel all fail.
Method testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
Methoud testCancelYarn failed because java.util.concurrent.TimeoutException:
Connection failed is located at WebFrontendITCase.java:324.
If maven test first runs testCancelYarn, its request to cancel fail, then
there's one job is running, so testCancel asserts no running job fail.
was (Author: lining):
Discover testCancelYarn and testCancel all fail.
Method testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
Methoud testCancelYarn failed because java.util.concurrent.TimeoutException:
Connection failed is located at WebFrontendITCase.java:324.
> WebFrontendITCase.testCancelYarn fails on travis
>
>
> Key: FLINK-15170
> URL: https://issues.apache.org/jira/browse/FLINK-15170
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Web Frontend
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: lining
>Priority: Major
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/622756846/log.txt
> {code}
> 23:59:33.299 [INFO] Running
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:33.365 [INFO] Running
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:35.379 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 2.012 s - in
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:38.802 [ERROR] Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time
> elapsed: 5.5 s <<< FAILURE! - in
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:38.802 [ERROR]
> testCancelYarn(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.436 s <<< ERROR!
> java.util.concurrent.TimeoutException: Connection failed
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancelYarn(WebFrontendITCase.java:324)
> 23:59:38.802 [ERROR]
> testCancel(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.003 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancel(WebFrontendITCase.java:240)
> 23:59:39.193 [INFO]
> 23:59:39.193 [INFO] Results:
> 23:59:39.193 [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15170) WebFrontendITCase.testCancelYarn fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005207#comment-17005207
]
lining edited comment on FLINK-15170 at 12/30/19 8:53 AM:
--
Discover testCancelYarn and testCancel all fail.
Method testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
Methoud testCancelYarn failed because java.util.concurrent.TimeoutException:
Connection failed is located at WebFrontendITCase.java:324.
was (Author: lining):
Discover testCancelYarn and testCancel all fail.
testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
testCancelYarn failed because java.util.concurrent.TimeoutException: Connection
failed is located at WebFrontendITCase.java:324.
> WebFrontendITCase.testCancelYarn fails on travis
>
>
> Key: FLINK-15170
> URL: https://issues.apache.org/jira/browse/FLINK-15170
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Web Frontend
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: lining
>Priority: Major
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/622756846/log.txt
> {code}
> 23:59:33.299 [INFO] Running
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:33.365 [INFO] Running
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:35.379 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 2.012 s - in
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:38.802 [ERROR] Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time
> elapsed: 5.5 s <<< FAILURE! - in
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:38.802 [ERROR]
> testCancelYarn(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.436 s <<< ERROR!
> java.util.concurrent.TimeoutException: Connection failed
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancelYarn(WebFrontendITCase.java:324)
> 23:59:38.802 [ERROR]
> testCancel(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.003 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancel(WebFrontendITCase.java:240)
> 23:59:39.193 [INFO]
> 23:59:39.193 [INFO] Results:
> 23:59:39.193 [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15170) WebFrontendITCase.testCancelYarn fails on travis
[
https://issues.apache.org/jira/browse/FLINK-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005207#comment-17005207
]
lining commented on FLINK-15170:
Discover testCancelYarn and testCancel all fail.
testCancel failed because java.lang.AssertionError is at
WebFrontendITCase.java:240.
testCancelYarn failed because java.util.concurrent.TimeoutException: Connection
failed is located at WebFrontendITCase.java:324.
> WebFrontendITCase.testCancelYarn fails on travis
>
>
> Key: FLINK-15170
> URL: https://issues.apache.org/jira/browse/FLINK-15170
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Web Frontend
>Affects Versions: 1.10.0
>Reporter: Dawid Wysakowicz
>Assignee: lining
>Priority: Major
> Labels: test-stability
> Fix For: 1.10.0
>
>
> https://api.travis-ci.org/v3/job/622756846/log.txt
> {code}
> 23:59:33.299 [INFO] Running
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:33.365 [INFO] Running
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:35.379 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 2.012 s - in
> org.apache.flink.runtime.webmonitor.handlers.JarSubmissionITCase
> 23:59:38.802 [ERROR] Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time
> elapsed: 5.5 s <<< FAILURE! - in
> org.apache.flink.runtime.webmonitor.WebFrontendITCase
> 23:59:38.802 [ERROR]
> testCancelYarn(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.436 s <<< ERROR!
> java.util.concurrent.TimeoutException: Connection failed
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancelYarn(WebFrontendITCase.java:324)
> 23:59:38.802 [ERROR]
> testCancel(org.apache.flink.runtime.webmonitor.WebFrontendITCase) Time
> elapsed: 0.003 s <<< FAILURE!
> java.lang.AssertionError
> at
> org.apache.flink.runtime.webmonitor.WebFrontendITCase.testCancel(WebFrontendITCase.java:240)
> 23:59:39.193 [INFO]
> 23:59:39.193 [INFO] Results:
> 23:59:39.193 [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15423) Show the detail of submitting job fail
[ https://issues.apache.org/jira/browse/FLINK-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15423: --- Description: Now when the user fails to submit the job, it just returns: unable to load the requested file. Could we return the details of the failure? (was: Now when the user fails to submit the job, it just returns: unable to load the requested file. Could we return details of the failure?) > Show the detail of submitting job fail > -- > > Key: FLINK-15423 > URL: https://issues.apache.org/jira/browse/FLINK-15423 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: lining >Priority: Major > > Now when the user fails to submit the job, it just returns: unable to load > the requested file. Could we return the details of the failure? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15423) Show the detail of submitting job fail
lining created FLINK-15423: -- Summary: Show the detail of submitting job fail Key: FLINK-15423 URL: https://issues.apache.org/jira/browse/FLINK-15423 Project: Flink Issue Type: Improvement Components: Runtime / REST Reporter: lining Now when the user fails to submit the job, it just returns: unable to load the requested file. Could we return details of the failure? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15422) Expose Jvm Metaspace、Compressed Class Space、Eden Space、Survivor Space、Old Gen metric
lining created FLINK-15422: -- Summary: Expose Jvm Metaspace、Compressed Class Space、Eden Space、Survivor Space、Old Gen metric Key: FLINK-15422 URL: https://issues.apache.org/jira/browse/FLINK-15422 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Reporter: lining Current, we just expose JVM the metrics of heap and non-heap. But users couldn't get details of JVM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15369) MiniCluster use fixed network / managed memory sizes by default
[ https://issues.apache.org/jira/browse/FLINK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15369: --- Summary: MiniCluster use fixed network / managed memory sizes by default (was: MiniCluster use fixed network / managed memory sizes by defualt) > MiniCluster use fixed network / managed memory sizes by default > --- > > Key: FLINK-15369 > URL: https://issues.apache.org/jira/browse/FLINK-15369 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Blocker > Fix For: 1.10.0 > > > Currently, Mini Cluster may allocate off-heap memory (managed & network) > according to the JVM free heap size and configured off-heap fractions. This > could lead to unnecessary large off-heap memory usage and unpredictable / > hard-to-understand behaviors. > We believe a fix value for managed / network memory would be enough for a > such a setup that runs Flink as a library. > Detailed discussion can be found in this [ML > thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15345) CurrentExecution and priorExecutions are inconsistent in the test case
[ https://issues.apache.org/jira/browse/FLINK-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000780#comment-17000780 ] lining commented on FLINK-15345: Referring to the [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionVertex.java#L633-L638], the prior execution will be put in the prior executions when restarting. Then it will create a new execution whose attempt number is the prior execution's attempt number plus one. As the code link is shown in the description, if the attempt number of current execution is 2, there should be two previous attempts in prior executions. > CurrentExecution and priorExecutions are inconsistent in the test case > -- > > Key: FLINK-15345 > URL: https://issues.apache.org/jira/browse/FLINK-15345 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST, Tests >Reporter: lining >Priority: Minor > > When the length of priorExecutions does not exceed the value of the > configuration item jobmanager.execution.attempts-history-size, the attempt > should be equal to the length of priorExecutions. So [code in > JobExceptionsHandlerTest|https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/JobExceptionsHandlerTest.java#L104-L121] > and [code in > SubtaskCurrentAttemptDetailsHandlerTest|https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/SubtaskCurrentAttemptDetailsHandlerTest.java#L94-L111] > are wrong, we need to update the attempt to 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15345) CurrentExecution and priorExecutions are inconsistent in the test case
lining created FLINK-15345: -- Summary: CurrentExecution and priorExecutions are inconsistent in the test case Key: FLINK-15345 URL: https://issues.apache.org/jira/browse/FLINK-15345 Project: Flink Issue Type: Improvement Components: Runtime / REST, Tests Reporter: lining When the length of priorExecutions does not exceed the value of the configuration item jobmanager.execution.attempts-history-size, the attempt should be equal to the length of priorExecutions. So [code in JobExceptionsHandlerTest|https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/JobExceptionsHandlerTest.java#L104-L121] and [code in SubtaskCurrentAttemptDetailsHandlerTest|https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/SubtaskCurrentAttemptDetailsHandlerTest.java#L94-L111] are wrong, we need to update the attempt to 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15329) Incorrect comment for MemoryManager#availableMemory
[ https://issues.apache.org/jira/browse/FLINK-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1676#comment-1676 ] lining commented on FLINK-15329: cc [~sewen] > Incorrect comment for MemoryManager#availableMemory > --- > > Key: FLINK-15329 > URL: https://issues.apache.org/jira/browse/FLINK-15329 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: lining >Priority: Minor > > As the > [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L672-L677] > shows, here is available memory, but the comment is total. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15329) Incorrect comment for MemoryManager#availableMemory
[ https://issues.apache.org/jira/browse/FLINK-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-15329: --- Description: As the [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L672-L677] shows, here is available memory, but the comment is total. (was: As the [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L672-L677] shows, here is available memory, but the comment is total.) > Incorrect comment for MemoryManager#availableMemory > --- > > Key: FLINK-15329 > URL: https://issues.apache.org/jira/browse/FLINK-15329 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: lining >Priority: Major > > As the > [code|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L672-L677] > shows, here is available memory, but the comment is total. -- This message was sent by Atlassian Jira (v8.3.4#803005)
