[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334125#comment-17334125 ] Flink Jira Bot commented on FLINK-10002: This issue was marked "stale-assigned" and has not received an update in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Xinyu Zhang >Assignee: Xinyu Zhang >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323500#comment-17323500 ] Flink Jira Bot commented on FLINK-10002: This issue is assigned but has not received an update in 7 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Xinyu Zhang >Assignee: Xinyu Zhang >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920723#comment-16920723 ] lining commented on FLINK-10002: [~zhangxinyu] Do you still work on it？ > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744944#comment-16744944 ] zhangxinyu commented on FLINK-10002: [~lining] Did you test it? If the log file length is smaller than the input number, it's expected to return the data from offset 0 to the length of the log file. Could you please show me the error? > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16716809#comment-16716809 ] lining commented on FLINK-10002: [~zhangxinyu] have you had test read by page if count > 102400, web will be error which is content length is not match. > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669860#comment-16669860 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 opened a new pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770 ## What is the purpose of the change This commit makes WebUI show jm/tm logs more friendly. WebUI can show historical logs by inputing a filename and a offset range to read of the file, shown as follows.     This pr adds 6 new REST APIs: - For JobManager `/jobmanager/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/jobmanager/loglist`Return the list of historical log filename. `/jobmanager/stdout?start=0&size=10240`Return the stdout data in the range - For TaskManager `/taskmanagers/:taskmanagerid/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/taskmanagers/:taskmanagerid/loglist`Return the list of historical log filename. `/taskmanagers/:taskmanagerid/stdout?start=0&size=10240`Return the stdout data in the range ## Brief change log This pr mainly add 6 APIs in new (WebMonitorEndpoint) REST API. Besides, some related frontend codes are changed, such as 1. `JobManagerLogFileHandler` is used to handle the request to fetch stdout and log file of job manager. The related headers are `JobManagerLogFileHeaders` and `JobManagerStdoutFileHeaders`. 2. `JobManagerLogListHandler` is used to handle the request to fetch log list of job manager. The related header is `JobManagerLogListHeaders`, and return a `LogListInfo` value. 3. `TaskManagerLogHandler` is used to handle the request to fetch stdout and log file of a given task manager. The related headers are `TaskManagerLogFileHeaders` and `TaskManagerStdoutFileHeaders`. 4. `TaskManagerLogListHandler` is used to handle the request to fetch log list of a given task manager. The related header is `TaskManagerLogListHeaders`. 5. `FileOffsetRange` is designed to identify a range of log files, which includes a start offset and an end offset. `LogFilenameQueryParameter`, `LogSizeQueryParameter` and `LogStartOffsetQueryParameter` are the parameters of requests. ## Verifying this change No tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669857#comment-16669857 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-434628064 > I think we missed an important thing. We should offer file size to user. If not, the parameters 'start', 'end' or 'size' would be useless since the user may have no idea how to set these parameters. Maybe we can set the start=0 and size=fileSize as the default value when user choose a log file. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669799#comment-16669799
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r229611970
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/FileOffsetRange.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1875#comment-1875 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-433834252 > About the test, I don't find any tests for the REST API of logs. Should I add tests for `JobManagerLogFileHandler`, `JobManagerLogListHandler`, `TaskManagerLogListHandler ` and `TaskManagerLogFileHandler `? Yes, you should add some unit tests to make sure the basic logic is right. Some old codes may not be covered well by testing. However the newer codes are covered quite well. We should keep this good habit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1866#comment-1866
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r228834689
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##
@@ -834,10 +843,20 @@ public void heartbeatFromResourceManager(ResourceID
resourceID) {
final File file = new File(filePath);
if (file.exists()) {
+ final RandomAccessFile raf;
+ try {
+raf = new RandomAccessFile(file, "r");
+ }catch (FileNotFoundException e) {
Review comment:
```suggestion
} catch (FileNotFoundException e) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1868#comment-1868
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r228833756
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/FileOffsetRange.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
Review comment:
We should implement the `hashCode` and `equals` methods. Because the
`FileOffsetRange` would be used in cache key. These methods would be used to
decide whether the query hit the cache or not. If these methods are missing,
the cache would be always missing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1867#comment-1867
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r228834637
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##
@@ -814,14 +819,18 @@ public void heartbeatFromResourceManager(ResourceID
resourceID) {
}
@Override
- public CompletableFuture requestFileUpload(FileType
fileType, Time timeout) {
+ public CompletableFuture requestFileUpload(FileType
fileType, Time timeout, String filename, FileOffsetRange range) {
log.debug("Request file {} upload.", fileType);
final String filePath;
switch (fileType) {
case LOG:
- filePath =
taskManagerConfiguration.getTaskManagerLogPath();
+ if(filename != null && !filename.equals("")) {
Review comment:
```suggestion
if (filename != null && !filename.equals("")) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663171#comment-16663171 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-432905856 @ifndef-SleePy Thanks for your suggestions. I have modified the code. I removed the legacy part and refactor the implementation. About the test, I don't find any tests for the REST API of logs. Should I add tests for `JobManagerLogFileHandler`, `JobManagerLogListHandler`, `TaskManagerLogListHandler ` and `TaskManagerLogFileHandler `? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660054#comment-16660054
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r227217598
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/RangePathParameter.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.messages;
+
+import org.apache.flink.runtime.rest.handler.legacy.files.FileOffsetRange;
+
+/**
+ * Path parameter identifying the offset range of a log file to read.
+ */
+public class RangePathParameter extends MessagePathParameter {
Review comment:
`/taskmanagers/:%s/log?filename=xxx&start=123&size=456` may be better since
it is compatible with the old API `/taskmanagers/:%s/log`, in which `filename`,
`start` and `size` are all optional.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660028#comment-16660028
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r227212699
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/RangePathParameter.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.messages;
+
+import org.apache.flink.runtime.rest.handler.legacy.files.FileOffsetRange;
+
+/**
+ * Path parameter identifying the offset range of a log file to read.
+ */
+public class RangePathParameter extends MessagePathParameter {
Review comment:
Good idea!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660009#comment-16660009
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r227210681
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerLogListHandler.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.taskmanager;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerGateway;
+import org.apache.flink.runtime.rest.AbstractHandler;
+import org.apache.flink.runtime.rest.handler.HandlerRequest;
+import org.apache.flink.runtime.rest.handler.RestHandlerException;
+import org.apache.flink.runtime.rest.handler.util.HandlerUtils;
+import org.apache.flink.runtime.rest.handler.util.KeepAliveWrite;
+import org.apache.flink.runtime.rest.messages.ErrorResponseBody;
+import org.apache.flink.runtime.rest.messages.UntypedResponseMessageHeaders;
+import
org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerIdPathParameter;
+import org.apache.flink.runtime.util.JsonUtils;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.Unpooled;
+import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.DefaultFullHttpResponse;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpHeaders;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpRequest;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpVersion;
+
+import javax.annotation.Nonnull;
+
+import java.nio.charset.Charset;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * TaskManagerLogListHandler serves the request which gets the historical log
file list of a given taskmanager.
+ */
+public class TaskManagerLogListHandler extends AbstractHandler {
Review comment:
Yes, you'r right. I'll modify it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660007#comment-16660007
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r227210230
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/taskmanager/TaskManagerLogFileWithRangeHeaders.java
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.messages.taskmanager;
+
+import org.apache.flink.runtime.rest.HttpMethodWrapper;
+import
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.runtime.rest.messages.RangePathParameter;
+import org.apache.flink.runtime.rest.messages.UntypedResponseMessageHeaders;
+
+import java.util.Arrays;
+
+/**
+ * Headers for the {@link TaskManagerLogFileHandler}.
+ */
+public class TaskManagerLogFileWithRangeHeaders implements
UntypedResponseMessageHeaders {
Review comment:
I'm wondering should the current implementation be compatible with previous
interfaces `/taskmanagers/:taskmanagerId/log` and
`/taskmanagers/:taskmanagerId/stdout`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660003#comment-16660003
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r227209341
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerStdoutFileHandler.java
##
@@ -54,7 +55,7 @@ public TaskManagerStdoutFileHandler(
}
@Override
- protected CompletableFuture
requestFileUpload(ResourceManagerGateway resourceManagerGateway, ResourceID
taskManagerResourceId) {
- return
resourceManagerGateway.requestTaskManagerFileUpload(taskManagerResourceId,
FileType.STDOUT, timeout);
+ protected CompletableFuture
requestFileUpload(ResourceManagerGateway resourceManagerGateway, ResourceID
taskManagerResourceId, String filename, FileOffsetRange range) {
Review comment:
".out" filename wouldn't be in log list. The log list only includes
filename of all historical logs. If we modify the implement of
`LogListHandler`, we can also get the stdout filename. However, do you think we
should mix them up?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659969#comment-16659969
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226959326
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659965#comment-16659965
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226958244
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659967#comment-16659967
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226958660
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659966#comment-16659966
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226958285
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659299#comment-16659299
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226959326
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659291#comment-16659291
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226958244
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659292#comment-16659292
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226958285
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
The RESTful API like "/xxx/taskmanager/$filename?start=123&size=456" is
better to use. However, I think 'Range' is always understood as something from
'start' to 'end'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658803#comment-16658803
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226934455
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StaticFileServerHandler.java
##
@@ -104,6 +106,8 @@
/** Be default, we allow files to be cached for 5 minutes. */
private static final int HTTP_CACHE_SECONDS = 300;
+ private static final Pattern currentLogPattern =
Pattern.compile("^/jobmanager/(log|stdout)(/([^/]*))?$");
Review comment:
Oh that's my fault. We still need this in master side.
Ignore my comment upside.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658802#comment-16658802
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226235686
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StaticFileServerHandler.java
##
@@ -128,30 +132,39 @@ public StaticFileServerHandler(
protected void respondAsLeader(ChannelHandlerContext
channelHandlerContext, RoutedRequest routedRequest, T gateway) throws Exception
{
final HttpRequest request = routedRequest.getRequest();
final String requestPath;
+ final FileOffsetRange range;
// make sure we request the "index.html" in case there is a
directory request
if (routedRequest.getPath().endsWith("/")) {
requestPath = routedRequest.getPath() + "index.html";
+ range = FileOffsetRange.MAX_FILE_OFFSET_RANGE;
+ } else if
(historicalLogPattern.matcher(routedRequest.getPath()).find()) {
+ Matcher m =
historicalLogPattern.matcher(routedRequest.getPath());
+ m.find();
+ requestPath = m.group(1);
+ range = FileOffsetRange.generateRange(m.group(2));
}
// in case the files being accessed are logs or stdout files,
find appropriate paths.
- else if (routedRequest.getPath().equals("/jobmanager/log") ||
routedRequest.getPath().equals("/jobmanager/stdout")) {
Review comment:
We should remove these log related codes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658801#comment-16658801
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226235403
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StaticFileServerHandler.java
##
@@ -104,6 +106,8 @@
/** Be default, we allow files to be cached for 5 minutes. */
private static final int HTTP_CACHE_SECONDS = 300;
+ private static final Pattern currentLogPattern =
Pattern.compile("^/jobmanager/(log|stdout)(/([^/]*))?$");
Review comment:
I think we should not use `StaticFileServerHandler` to process log file
anymore.
The log request would be served by other log handler.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656409#comment-16656409
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226159648
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServer.java
##
@@ -533,6 +533,11 @@ public TransientBlobKey putTransient(InputStream
inputStream) throws IOException
return (TransientBlobKey) putInputStream(null, inputStream,
TRANSIENT_BLOB);
}
+ @Override
+ public TransientBlobKey putTransient(InputStream inputStream, long
count) throws IOException {
Review comment:
Same with `TransientBlobService.java`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656393#comment-16656393
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226523510
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
+ private final long end;
+
+ private FileOffsetRange(long start, long end) {
+ this.start = start;
+ this.end = end;
+ }
+
+ public static final FileOffsetRange MAX_FILE_OFFSET_RANGE = new
FileOffsetRange(0, Long.MAX_VALUE);
+
+ public static FileOffsetRange generateRange(String range) {
Review comment:
Let's put the format codes in another place. `FileOffsetRange` would be used
in `TaskExecutor/ResourceManager`. They have nothing related with the format in
RESTful API.
`FileOffsetRange` should be quite clean. Just has two properties, start and
size.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656404#comment-16656404 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226223893 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/LogListHandler.java ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.rest.handler.legacy.files; Review comment: Legacy problem again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656383#comment-16656383 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226158300 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/TransientBlobService.java ## @@ -123,6 +123,21 @@ */ TransientBlobKey putTransient(InputStream inputStream) throws IOException; + /** +* Uploads a give size of (job-unrelated) data from the given input stream to the BLOB server. +* +* @param inputStream +* the input stream to read the data from +* @param count the size of data to upload to blob server +* +* @return the computed BLOB key identifying the BLOB on the server +* +* @throws IOException +* thrown if an I/O error occurs while reading the data from the input stream or uploading the +* data to the BLOB server +*/ + TransientBlobKey putTransient(InputStream inputStream, long count) throws IOException; Review comment: Actually `InputStream` is enough here. We can put an input stream that representing partial of the log file. See [`BytesStream.limit() in guava`](https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/ByteStreams.html#limit(java.io.InputStream,%20long)) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656382#comment-16656382 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226154751 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ## @@ -373,6 +373,54 @@ BlobKey putInputStream(@Nullable JobID jobId, InputStream inputStream, BlobKey.B } } + /** +* Uploads a given size of data from the given input stream to the BLOB server. +* +* @param jobId +* the ID of the job the BLOB belongs to (or null if job-unrelated) +* @param inputStream +* the input stream to read the data from +* @param blobType +* whether the BLOB should become permanent or transient +* @return the computed BLOB key of the uploaded BLOB +* +* @param count the size of data to upload +* +* @return the computed BLOB key of the uploaded BLOB +* +* @throws IOException +* thrown if an I/O error occurs while uploading the data to the BLOB server +*/ + BlobKey putInputStream (@Nullable JobID jobId, InputStream inputStream, BlobKey.BlobType blobType, long count) Review comment: ```suggestion BlobKey putInputStream(@Nullable JobID jobId, InputStream inputStream, BlobKey.BlobType blobType, long count) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656401#comment-16656401
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226522012
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##
@@ -622,7 +628,19 @@ public void unRegisterInfoMessageListener(final String
address) {
log.debug("Requested file {} upload from unregistered
TaskExecutor {}.", fileType, taskManagerId);
return FutureUtils.completedExceptionally(new
UnknownTaskExecutorException(taskManagerId));
} else {
- return
taskExecutor.getTaskExecutorGateway().requestFileUpload(fileType, timeout);
+ return
taskExecutor.getTaskExecutorGateway().requestFileUpload(fileType, timeout,
filename, range);
+ }
+ }
+
+ @Override
+ public CompletableFuture requestTaskManagerLogList(ResourceID
taskManagerId, Time timeout){
Review comment:
```suggestion
public CompletableFuture requestTaskManagerLogList(ResourceID
taskManagerId, Time timeout) {
```
Style problem
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656399#comment-16656399
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226545493
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerStdoutFileHandler.java
##
@@ -54,7 +55,7 @@ public TaskManagerStdoutFileHandler(
}
@Override
- protected CompletableFuture
requestFileUpload(ResourceManagerGateway resourceManagerGateway, ResourceID
taskManagerResourceId) {
- return
resourceManagerGateway.requestTaskManagerFileUpload(taskManagerResourceId,
FileType.STDOUT, timeout);
+ protected CompletableFuture
requestFileUpload(ResourceManagerGateway resourceManagerGateway, ResourceID
taskManagerResourceId, String filename, FileOffsetRange range) {
Review comment:
Actually we don't need this handler anymore.
We can treat stdout as a normal log file since we have log listing handlers.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656390#comment-16656390
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226235686
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StaticFileServerHandler.java
##
@@ -128,30 +132,39 @@ public StaticFileServerHandler(
protected void respondAsLeader(ChannelHandlerContext
channelHandlerContext, RoutedRequest routedRequest, T gateway) throws Exception
{
final HttpRequest request = routedRequest.getRequest();
final String requestPath;
+ final FileOffsetRange range;
// make sure we request the "index.html" in case there is a
directory request
if (routedRequest.getPath().endsWith("/")) {
requestPath = routedRequest.getPath() + "index.html";
+ range = FileOffsetRange.MAX_FILE_OFFSET_RANGE;
+ } else if
(historicalLogPattern.matcher(routedRequest.getPath()).find()) {
+ Matcher m =
historicalLogPattern.matcher(routedRequest.getPath());
+ m.find();
+ requestPath = m.group(1);
+ range = FileOffsetRange.generateRange(m.group(2));
}
// in case the files being accessed are logs or stdout files,
find appropriate paths.
- else if (routedRequest.getPath().equals("/jobmanager/log") ||
routedRequest.getPath().equals("/jobmanager/stdout")) {
Review comment:
We should remove these log related codes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656405#comment-16656405 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226519814 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/slots/TaskManagerGateway.java ## @@ -179,15 +180,23 @@ void triggerCheckpoint( * @param timeout for the request * @return Future blob key under which the task manager log has been stored */ - CompletableFuture requestTaskManagerLog(final Time timeout); + CompletableFuture requestTaskManagerLog(final Time timeout, String filename, FileOffsetRange range); Review comment: TaskManager related codes should be abandoned in future. We can just leave this alone. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656384#comment-16656384
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226159648
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServer.java
##
@@ -533,6 +533,11 @@ public TransientBlobKey putTransient(InputStream
inputStream) throws IOException
return (TransientBlobKey) putInputStream(null, inputStream,
TRANSIENT_BLOB);
}
+ @Override
+ public TransientBlobKey putTransient(InputStream inputStream, long
count) throws IOException {
Review comment:
Same with `TransientBlobService.java'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656394#comment-16656394
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226523853
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/LogFileHandlerSpecification.java
##
@@ -25,12 +25,18 @@
* Rest handler specification for the log file of the main cluster component.
*/
public class LogFileHandlerSpecification implements RestHandlerSpecification {
-
- private static final LogFileHandlerSpecification INSTANCE = new
LogFileHandlerSpecification();
-
private static final String URL = "/jobmanager/log";
+ private static final String URL_WITH_RANGE = "/jobmanager/log/:range";
+ private static final String URL_WITH_FILENAME_AND_RANGE =
"/jobmanager/logs/:filename/:range";
- private LogFileHandlerSpecification() {}
+ private static final LogFileHandlerSpecification INSTANCE = new
LogFileHandlerSpecification(URL);
Review comment:
Singleton with multiple instances is not a good idea anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656389#comment-16656389
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226159805
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/TransientBlobCache.java
##
@@ -141,6 +141,13 @@ public TransientBlobKey putTransient(InputStream
inputStream) throws IOException
}
}
+ @Override
+ public TransientBlobKey putTransient(InputStream inputStream, long
count) throws IOException {
Review comment:
Same with `TransientBlobService.java`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656406#comment-16656406 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226549450 ## File path: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala ## @@ -338,6 +340,9 @@ class TaskManager( "available. Cannot upload TaskManager logs.")) } +case LogListRequest => Review comment: Legacy problem again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656403#comment-16656403
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226549357
##
File path:
flink-runtime/src/main/scala/org/apache/flink/runtime/messages/TaskManagerMessages.scala
##
@@ -117,11 +118,13 @@ object TaskManagerMessages {
/** Trait do differentiate which log file is requested */
sealed trait LogTypeRequest
- /** Indicates a request for the .log file */
- case object LogFileRequest extends LogTypeRequest
+ /** Indicates a request for the log file */
+ case class LogFileRequest(filename : String, range: FileOffsetRange) extends
LogTypeRequest
Review comment:
Just leave the legacy codes alone.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656408#comment-16656408
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226236068
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StdoutFileHandlerSpecification.java
##
@@ -25,12 +25,17 @@
* Rest handler specification for the stdout file of the main cluster
component.
*/
public class StdoutFileHandlerSpecification implements
RestHandlerSpecification {
Review comment:
Legacy problem again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656397#comment-16656397
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226521972
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##
@@ -622,7 +628,19 @@ public void unRegisterInfoMessageListener(final String
address) {
log.debug("Requested file {} upload from unregistered
TaskExecutor {}.", fileType, taskManagerId);
return FutureUtils.completedExceptionally(new
UnknownTaskExecutorException(taskManagerId));
} else {
- return
taskExecutor.getTaskExecutorGateway().requestFileUpload(fileType, timeout);
+ return
taskExecutor.getTaskExecutorGateway().requestFileUpload(fileType, timeout,
filename, range);
+ }
+ }
+
+ @Override
+ public CompletableFuture requestTaskManagerLogList(ResourceID
taskManagerId, Time timeout){
+ log.debug("Request historical loglist from TaskExecutor {}.",
taskManagerId);
Review comment:
Log would be too much, I think we should avoid debug log in normal code path.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656402#comment-16656402
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226546566
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/taskmanager/TaskManagerLogFileWithRangeHeaders.java
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.messages.taskmanager;
+
+import org.apache.flink.runtime.rest.HttpMethodWrapper;
+import
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.runtime.rest.messages.RangePathParameter;
+import org.apache.flink.runtime.rest.messages.UntypedResponseMessageHeaders;
+
+import java.util.Arrays;
+
+/**
+ * Headers for the {@link TaskManagerLogFileHandler}.
+ */
+public class TaskManagerLogFileWithRangeHeaders implements
UntypedResponseMessageHeaders {
Review comment:
We should unify this one with `TaskManagerLogFileHeaders`,
`TaskManagerHistoricalLogFileHeaders` and `TaskManagerStdoutFileHeaders`.
`/taskmanagers/:taskmanagerId/logs/:logFilename?:range` is enough for all
these.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656407#comment-16656407
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226525650
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/RangePathParameter.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.messages;
+
+import org.apache.flink.runtime.rest.handler.legacy.files.FileOffsetRange;
+
+/**
+ * Path parameter identifying the offset range of a log file to read.
+ */
+public class RangePathParameter extends MessagePathParameter {
Review comment:
I prefer treat 'range' as a `MessageQueryParameter`.
It would be better that range is an optional parameter. If there is no range
parameter, we can just return the whole log file. It is compatible with the old
API.
The RESTful API should be like
"/xxx/taskmanager/$filename?start=123&size=456"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656392#comment-16656392
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226235403
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/StaticFileServerHandler.java
##
@@ -104,6 +106,8 @@
/** Be default, we allow files to be cached for 5 minutes. */
private static final int HTTP_CACHE_SECONDS = 300;
+ private static final Pattern currentLogPattern =
Pattern.compile("^/jobmanager/(log|stdout)(/([^/]*))?$");
Review comment:
I think we should not use `StaticFileServerHandler` to process log file
anymore.
The log request would be served by other log handler.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656388#comment-16656388 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226522210 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.rest.handler.legacy.files; Review comment: I don't think we should put the class in this package. 1. It's under legacy package. It may be abandoned later. 2. It's quite a general class. Not just for RESTful API. TaskExecutor/ResourceManager should not depend on any RESTful related codes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656396#comment-16656396
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226549245
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/JsonUtils.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import org.apache.flink.runtime.rest.handler.legacy.JsonFactory;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.util.FlinkException;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Comparator;
+import java.util.TreeSet;
+import java.util.concurrent.CompletionException;
+
+/**
+ * Convert collections to json string
+ */
+public class JsonUtils {
Review comment:
Try to use 'Jackson json framework' instead of constructing response
manually.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656391#comment-16656391
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226233769
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/LogListHandler.java
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.rest.handler.RedirectHandler;
+import org.apache.flink.runtime.rest.handler.router.RoutedRequest;
+import org.apache.flink.runtime.rest.handler.util.KeepAliveWrite;
+import org.apache.flink.runtime.util.JsonUtils;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.Unpooled;
+import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.DefaultFullHttpResponse;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpHeaders;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpVersion;
+
+import javax.annotation.Nonnull;
+
+import java.io.File;
+import java.nio.charset.Charset;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * LogListHandler serves the request which gets the historical log file list
of jobmanager.
+ */
+public class LogListHandler extends RedirectHandler {
+ private static final Charset ENCODING = Charset.forName("UTF-8");
+ private final File parentPath;
Review comment:
Style problem.
We should separate these lines with empty line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656398#comment-16656398
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226522992
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/FileOffsetRange.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.legacy.files;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
+ private static final String SEPARATOR = "-";
+ private final long start;
Review comment:
I prefer 'start + size', not 'start + end'. Size is more natural.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656400#comment-16656400
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226548116
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
##
@@ -814,14 +818,18 @@ public void heartbeatFromResourceManager(ResourceID
resourceID) {
}
@Override
- public CompletableFuture requestFileUpload(FileType
fileType, Time timeout) {
+ public CompletableFuture requestFileUpload(FileType
fileType, Time timeout, String filename, FileOffsetRange range) {
Review comment:
We should unify stdout and log file as aforementioned.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656386#comment-16656386 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226230533 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/LogListHandlerSpecification.java ## @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.rest.handler.legacy.files; Review comment: Legacy problem again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656395#comment-16656395
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226549005
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/taskmanager/TaskManagerLogListHandler.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.taskmanager;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerGateway;
+import org.apache.flink.runtime.rest.AbstractHandler;
+import org.apache.flink.runtime.rest.handler.HandlerRequest;
+import org.apache.flink.runtime.rest.handler.RestHandlerException;
+import org.apache.flink.runtime.rest.handler.util.HandlerUtils;
+import org.apache.flink.runtime.rest.handler.util.KeepAliveWrite;
+import org.apache.flink.runtime.rest.messages.ErrorResponseBody;
+import org.apache.flink.runtime.rest.messages.UntypedResponseMessageHeaders;
+import
org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerIdPathParameter;
+import org.apache.flink.runtime.util.JsonUtils;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.flink.shaded.netty4.io.netty.buffer.Unpooled;
+import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.DefaultFullHttpResponse;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpHeaders;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpRequest;
+import
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpVersion;
+
+import javax.annotation.Nonnull;
+
+import java.nio.charset.Charset;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * TaskManagerLogListHandler serves the request which gets the historical log
file list of a given taskmanager.
+ */
+public class TaskManagerLogListHandler extends AbstractHandler {
Review comment:
It's too low-level. See other handlers in `WebMonitorEndpoint`. Extends
`AbstractRestHandler` instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656387#comment-16656387
]
ASF GitHub Bot commented on FLINK-10002:
ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r226220834
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/files/LogFileHandlerSpecification.java
##
@@ -25,12 +25,18 @@
* Rest handler specification for the log file of the main cluster component.
*/
public class LogFileHandlerSpecification implements RestHandlerSpecification {
Review comment:
This class is under legacy package.
We should add the feature in a new file. Just leave these legacy codes alone.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656385#comment-16656385 ] ASF GitHub Bot commented on FLINK-10002: ifndef-SleePy commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#discussion_r226159561 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ## @@ -373,6 +373,54 @@ BlobKey putInputStream(@Nullable JobID jobId, InputStream inputStream, BlobKey.B } } + /** +* Uploads a given size of data from the given input stream to the BLOB server. +* +* @param jobId +* the ID of the job the BLOB belongs to (or null if job-unrelated) +* @param inputStream +* the input stream to read the data from +* @param blobType +* whether the BLOB should become permanent or transient +* @return the computed BLOB key of the uploaded BLOB +* +* @param count the size of data to upload +* +* @return the computed BLOB key of the uploaded BLOB +* +* @throws IOException +* thrown if an I/O error occurs while uploading the data to the BLOB server +*/ + BlobKey putInputStream (@Nullable JobID jobId, InputStream inputStream, BlobKey.BlobType blobType, long count) Review comment: Same problem with `TransientBlobService.java`. We can avoid this new method. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651721#comment-16651721 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-430243798 @yanghua Thanks. Conflits are resovled. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651256#comment-16651256 ] ASF GitHub Bot commented on FLINK-10002: yanghua commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-430130122 @zhangxinyu1 this PR has conflicting files, please update and rebase it. I will try to look around before Till. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641483#comment-16641483 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-427748392 @tillrohrmann Thanks, I'll resolve conflicts if it has until you review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640802#comment-16640802 ] ASF GitHub Bot commented on FLINK-10002: tillrohrmann commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-427592617 Thanks for your contribution @zhangxinyu1. From a high level perspective, this adds a good feature. At the moment I'm quite busy and it will take some time for me to review your PR. Please bear with us until then. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630168#comment-16630168 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-425044201 @buptljy Could you please take a look at the pr. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629898#comment-16629898 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-424996171 @yanghua @tillrohrmann Could you please review this pr? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629888#comment-16629888 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 opened a new pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770 ## What is the purpose of the change This commit makes WebUI show jm/tm logs more friendly. WebUI can show historical logs by inputing a filename and a offset range to read of the file, shown as follows.     This pr adds 8 new REST APIs: - For JobManager `/jobmanager/log/:range`Return logs of current log file in the range. `/jobmanager/logs/:filename/:range`Return logs of historical log file in the range. `/jobmanager/loglist`Return the list of historical log filename. `/jobmanager/stdout/:range`Return the stdout data in the range - For TaskManager `/taskmanagers/:taskmanagerid/log/:range`Return logs of current log file in the range. `/taskmanagers/:taskmanagerid/logs/:filename/:range` Return logs of historical log file in the range. `/taskmanagers/:taskmanagerid/loglist`Return the list of historical log filename. `/taskmanagers/:taskmanagerid/stdout/:range`Return the stdout data in the range ## Brief change log This pr mainly add 8 APIs in both legacy (WebRuntimeMonitor) and new (WebMonitorEndpoint) REST API. Besides, some related frontend codes are changed. - For legacy REST API: 1. In `WebRuntimeMonitor.java`, paths `/jobmanager/log/:range`, `/jobmanager/stdout/:range` and `/jobmanager/logs/:filename/:range` are handled by `StaticFileServerHandler`. Therefore, `StaticFileServerHandler.java` is also changed to handle these three paths. 2. `JobManagerLogListHandler` and `TaskManagerLogListHandler` are added to deal with the path `/jobmanager/loglist` and `/taskmanagers/:taskmanagerid/loglist` to get the historical log file names. Besides, a new method `requestTaskManagerLogList()` is added to `TaskManagerGateway` to send a new TaskManagerMessage `LogListRequest` to request taskmanager’s log file list. TaskManager deal with this request by a new method `handleRequestTaskManagerLogList()`. 3. There are three paths `/taskmanagers/:taskmanagerid/log/:range`, `/taskmanagers/:taskmanagerid/logs/:filename/:range` and `/taskmanagers/:taskmanagerid/stdout/:range` added to `TaskManagerLogHandler.java`. At the same time, `TaskManagerGateway`, `ActorTaskManagerGateWay`, `TaskManagerMessages` and `TaskManager` are changed to handle these three paths. 4. `FileOffsetRange.java` is designed to identify a range of log files, which includes a start offset and an end offset. - For new REST API: 1. In LogFileHandlerSpecification, paths `/jobmanager/log/:range` and `/jobmanager/logs/:filename/:range` are added. In `StdoutFileHandlerSpecification,` `/jobmanager/stdout/:range` is added. These three paths are also handled by `StaticFileServerHandler` which is same with the legacy REST API. 2. `LogListHandler` and `TaskManagerLogListHandler` are added to deal with the path `/jobmanager/loglist` and `/taskmanagers/:taskmanagerid/loglist`. This `TaskManagerLogListHandler` is different from the `TaskManagerLogListHandler` in legacy REST API. To get the historical log list, a new method `requestTaskManagerLogList()` is added to `ResourceManagerGateway`. Its related classes and interfaces, such as `ResourceManager`, `TaskExecutorGateway` and `TaskExecutor`, are also changed. 3. To deal with `/taskmanagers/:taskmanagerid/log/:range`, `/taskmanagers/:taskmanagerid/logs/:filename/:range` and `/taskmanagers/:taskmanagerid/stdout/:range`, `AbstractTaskManagerFileHandler` is changed. The mainly difference is that filename and range are also a part of cache key. When loading task manager file, filename and range are also parameters. ## Verifying this change This change added tests and can be verified as follows: - Added integration tests for all new paths in `WebFrontendITCase.java` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (p
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610059#comment-16610059 ] zhangxinyu commented on FLINK-10002: [~wind_ljy] Thanks for your suggestions. # We could use "SizeAndTimeBasedRollingPolicy" to split by size or time. I think batch jobs may also have a large mount of logs. As we know, if the mount of logs is too large, the log UI may crash, so we can split by size. # Yes, the current implementation wouldn't cache logs. I think if the log is rotated, the log wouldn't be changed, so we can cache them in blob server. > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Priority: Major > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610028#comment-16610028 ] Jiayi Liao commented on FLINK-10002: [~zhangxinyu] Thanks for your idea, but I don't think that we should change the default log configuration. 1. Flink can also handle batch jobs, which may not need "daily rolling log". 2. From what I see in the description, it seems that the blob service will copy the log file multiple times and blob cache can't be used if you change the time interval, right? > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Priority: Major > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16608680#comment-16608680 ] zhangxinyu commented on FLINK-10002: Hi [~trohrm...@apache.org], could you please assign this issue to me. I have finished some work for this issue: # Use DailyRoolingAppender to split logs by default. # All logs can be selected through webui Shown as follows. !image-2018-09-10-11-38-07-973.png! > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Priority: Major > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
