Hi Jamie Grier, Thank you for your reply, let me add some explanations to this design.
First of all, as stated in "Goal", it is mainly for the "Standalone" cluster model, although we have implemented it for Flink on YARN, this does not mean that we can't turn off this feature by means of options. It should be noted that the separation is basically based on the "log configuration file", it is very scalable and even allows users to define the log pattern of the configuration file (of course this is an extension feature, not mentioned in the design documentation). In fact, "multiple files are a special case of a single file", we can provide an option to keep it still the default behavior, it should be the scene you expect in the container. According to Flink's official 2016 adjustment report [1], users using the standalone mode are quite close to the yarn mode (unfortunately there is no data support in 2017). Although we mainly use Flink on Yarn now, we have used standalone in depth (close to the daily processing volume of 20 trillion messages). In this scenario, the user logs generated by different job's tasks are mixed together, and it is very difficult to locate the issue. Moreover, as we configure the log file scrolling policy, we have to log in to the server to view it. Therefore, we expect that for the same task manager, the user logs generated by the tasks from the same job can be distinguished. In addition, I have tried MDC technology, but it can not achieve the goal. The underlying Flink is log4j 1.x and logback. We need to be compatible with both frameworks at the same time, and we don't allow large-scale changes to the active code, and no sense to the user. Some other points: 1) Many of our users have experience using Storm and Spark, and they are more accustomed to that style in standalone mode; 2) We split the user log by Job, which will help to implement the "business log aggregation" feature based on the Job. Best, Vino [1]: https://www.ververica.com/blog/flink-user-survey-2016-part-1 Jamie Grier <jgr...@lyft.com.invalid> 于2019年3月1日周五 上午7:32写道： > I think maybe if I understood this correctly this design is going in the > wrong direction. The problem with Flink logging, when you are running > multiple jobs in the same TMs, is not just about separating out the > business level logging into separate files. The Flink framework itself > logs many things where there is clearly a single job in context but that > all ends up in the same log file and with no clear separation amongst the > log lines. > > Also, I don't think shooting to have multiple log files is a very good idea > either. It's common, especially on container-based deployments, that the > expectation is that a process (like Flink) logs everything to stdout and > the surrounding tooling takes care of routing that log data somewhere. I > think we should stick with that model and expect that there will be a > single log stream coming out of each Flink process. > > Instead, I think it would be better to enhance Flink's logging capability > such that the appropriate context can be added to each log line with the > exact format controlled by the end user. It might make sense to take a > look at MDC, for example, as a way to approach this. > > > On Thu, Feb 28, 2019 at 4:24 AM vino yang <yanghua1...@gmail.com> wrote: > > > Dear devs, > > > > Currently, for log output, Flink does not explicitly distinguish between > > framework logs and user logs. In Task Manager, logs from the framework > are > > intermixed with the user's business logs. In some deployment models, such > > as Standalone or YARN session, there are different task instances of > > different jobs deployed in the same Task Manager. It makes the log event > > flow more confusing unless the users explicitly use tags to distinguish > > them and it makes locating problems more difficult and inefficient. For > > YARN job cluster deployment model, this problem will not be very serious, > > but we still need to artificially distinguish between the framework and > the > > business log. Overall, we found that Flink's existing log model has the > > following problems: > > > > > > - > > > > Framework log and business log are mixed in the same log file. There > > is no way to make a clear distinction, which is not conducive to > problem > > location and analysis; > > - > > > > Not conducive to the independent collection of business logs; > > > > > > Therefore, we propose a mechanism to separate the framework and business > > log. It can split existing log files for Task Manager. > > > > Currently, it is associated with two JIRA issue: > > > > - > > > > FLINK-11202[1]: Split log file per job > > - > > > > FLINK-11782[2]: Enhance TaskManager log visualization by listing all > > log files for Flink web UI > > > > > > We have implemented and validated it in standalone and Flink on YARN (job > > cluster) mode. > > > > sketch 1: > > > > [image: flink-web-ui-taskmanager-log-files.png] > > > > sketch 2: > > [image: flink-web-ui-taskmanager-log-files-2.png] > > > > Design documentation : > > > https://docs.google.com/document/d/1TTYAtFoTWaGCveKDZH394FYdRyNyQFnVoW5AYFvnr5I/edit?usp=sharing > > > > Best, > > Vino > > > > [1]: https://issues.apache.org/jira/browse/FLINK-11202 > > [2]: https://issues.apache.org/jira/browse/FLINK-11782 > > >
