Hi Xinyu, Thank you for your clarification on "periodic reading". If Flink considers developing an API for reading logs, I think this is a good idea.
Regarding the problem of TM reading logs, your idea is good from a performance perspective. But Flink didn't provide any web services for the TM from the beginning. All the requests were passed through the JM proxy. Just because of the log read performance changes, there will be major changes to the architecture, and this will make the TM take on more responsibilities. Thanks, vino. 2018-07-30 17:34 GMT+08:00 Xinyu Zhang <[email protected]>: > Thanks for your reply. "periodic reading" means reading all logs in a > given time interval. For example, my logs is daily divided, I can get all > logs of yesterday through a parameter like '2018-07-29/2018-07-30'. > > TM which provides a web service to display information will lessen the > burden of jobmanager, especially when there are many taskManagers in the > flink cluster. > > > 2018年7月30日星期一，vino yang <[email protected]> 写道： > >> Hi Xinyu, >> >> This is indeed a problem. Especially when the amount of logs is large, it >> may even cause the UI to stall for a long time. The same is true for YRAN, >> and there is really no good way to do it at the moment. >> Thank you for your suggestion, do you mean "periodic reading" refers to >> full or incremental? If it is a full reading, I personally do not recommend >> this, which will increase the burden on JM and client, and it is >> appropriate to manually trigger it by the user. Maybe we can consider >> loading only a few logs at a time, such as incremental reads, paged >> displays, and so on. >> Regarding the second question, Flink did this because its TM does not >> provide any web services to display information, and the Web UI is >> currently bundled with JM. >> >> Thanks, vino. >> >> 2018-07-30 16:33 GMT+08:00 Xinyu Zhang <[email protected]>: >> >>> Hi all >>> >>> We use flink on yarn and flink version is 1.4. >>> >>> When a streaming job run for a long time, the webUI cannot show logs. >>> This may be becasue the log size is too large. >>> >>> However, if we use the DailyRollingAppender to divide logs (granularity >>> is `day`) in log4j.properties, we will never see the log of yesterday. >>> >>> Is there any ideas can make read logs easier? >>> >>> Maybe, we should add an interface that support for reading log by time >>> interval. Besides, when we get the taskmanager logs through webUI, >>> jobmanager can redirect to a URL of the taskmanager, which users can get >>> the logs directly (Just like MR task), other than downloading the logs from >>> taskmanager and then sending logs to users. >>> >>> Thanks! >>> >>> Xinyu Zhang >>> >>> >>> >>
