Hi Jie, The configs defined in the stack definitions are only the default values and can be changed by the end user when deploying the cluster. You can follow these steps to change the log4j properties from the UI without having to change the default. https://community.hortonworks.com/content/supportkb/49706/kafka-log-file-rotation-based-on-file-size.html
Regards Jayush From: "wang...@cmss.chinamobile.com" <wang...@cmss.chinamobile.com> Date: Tuesday, October 23, 2018 at 7:45 AM To: jluniya <jlun...@apache.org>, atkach <atk...@apache.org>, sumitmohant <sumitmoh...@apache.org>, afernandez <afernan...@apache.org> Cc: 姚磊 <yao...@cmss.chinamobile.com>, 王宝晗 <wangbao...@cmss.chinamobile.com> Subject: Greetings from China Mobile Developers Focusing on Ambari Community Dear Ambari PMC: We are developers from China Mobile and have been contributing a great deal of patches to the ambari community. We have been using ambari for managing clusters with one of which has the maximum 1500+ nodes，and we are plannig to apply ambari 2.7 to a cluster with nearly 5000 nodes. When using ambari to manage the cluster, we found that the nodes on which had KAFKA service running were inefficient and some of them even broke down because the /var disk was full of kafka logs. Looking deep into this problem, we found that in ambari the KAFKA service log4j which uses DailyRollingFileAppender has some properties that it does not support maxBackupIndex and maxFileSize, which results that the kafka logs’s constantly increasing and would never be removed. Fortunately, the log4j appender RollingFileAppender supports these properties. Therefore we changed the entire kafka log4j appender from DailyRollingFileAppender to RollingFileAppender and the problem was solved. According to the ambari dev mailing list, we know that the community is going to do a short follow up maintenance release Apache Ambari 2.7.3, we hope our patch Ambari-20390 would be contributed to the lastest release version. And we are looking forward to the release of ambari 2.7.3 so that we can use it to manage our 5000 nodes cluster. Best Regards, Jie Wang China Moible
