Hi Xinyu, This is indeed a problem. Especially when the amount of logs is large, it may even cause the UI to stall for a long time. The same is true for YRAN, and there is really no good way to do it at the moment. Thank you for your suggestion, do you mean "periodic reading" refers to full or incremental? If it is a full reading, I personally do not recommend this, which will increase the burden on JM and client, and it is appropriate to manually trigger it by the user. Maybe we can consider loading only a few logs at a time, such as incremental reads, paged displays, and so on. Regarding the second question, Flink did this because its TM does not provide any web services to display information, and the Web UI is currently bundled with JM.
Thanks, vino. 2018-07-30 16:33 GMT+08:00 Xinyu Zhang <wsz...@gmail.com>: > Hi all > > We use flink on yarn and flink version is 1.4. > > When a streaming job run for a long time, the webUI cannot show logs. This > may be becasue the log size is too large. > > However, if we use the DailyRollingAppender to divide logs (granularity is > `day`) in log4j.properties, we will never see the log of yesterday. > > Is there any ideas can make read logs easier? > > Maybe, we should add an interface that support for reading log by time > interval. Besides, when we get the taskmanager logs through webUI, > jobmanager can redirect to a URL of the taskmanager, which users can get > the logs directly (Just like MR task), other than downloading the logs from > taskmanager and then sending logs to users. > > Thanks! > > Xinyu Zhang > > >
