Hello Navneet Kumar Pandey, org.apache.log4j.rolling.RollingFileAppender is part of Apache Extras Companion for Apache log4j [1]. Is that library in your classpath?
Are there hints in taskmanager.err? Can you run: cat /usr/lib/flink/conf/log4j.properties on the EMR master node and show the output? For troubleshooting, you can also try org.apache.log4j.RollingFileAppender which can roll the file if a certain size is exceeded. An example configuration can be found here (I have not tested it): https://github.com/apache/flink/pull/5371/files Best, Gary [1] https://logging.apache.org/log4j/extras/ On Fri, Aug 17, 2018 at 4:09 PM, Navneet Kumar Pandey <navn...@essens.no> wrote: > I am using Flink in EMR with following configuration. > > { > "Classification": "flink-log4j", > "Properties": { > "log4j.logger.no":"DEBUG", > "log4j.appender.file":"org.apache.log4j.rolling. > RollingFileAppender", > "log4j.appender.file.RollingPolicy.FileNamePattern" > :"logs/log.%d{yyyyMMdd-HHmm}.log", > "log4j.appender.file.RollingPolicy":"org.apache.log4j.rolling. > TimeBasedRollingPolicy", > "log4j.appender.file.append":"false", > "log4j.appender.file.layout":"org.apache.log4j.PatternLayout", > "log4j.appender.file.layout.ConversionPattern":"%d{yyyy-MM-dd > HH:mm:ss,SSS} %-5p %-60c %x - %m%n" > > } > } > > FYI this configuration get written into flink's log4j.properties.As you > can see even after this setting taskmanager and jobmanager log files are > not getting rolled. > > [hadoop@ip-XXXXXX ~]$ sudo ls -lh /mnt/var/log/hadoop-yarn/ > containers/application_DDDDDDDDDDD_0002/container_ > DDDDDDDDDDDD_0002_01_000002 > total 7.0G > -rw-r--r-- 1 yarn yarn 770K Aug 17 14:02 taskmanager.err > -rw-r--r-- 1 yarn yarn 6.0G Aug 17 14:02 taskmanager.log > -rw-r--r-- 1 yarn yarn 526K Aug 17 13:54 taskmanager.out > > Can somebody help me to give pointer about how to roll these log files? > Note that these files are also being copied into s3. > >