Ah I see. I think the Flink app is reading files from /opt/flink/conf correctly as it is, since changes I make to flink-conf are picked up as expected, it's just the log4j properties that are either not being used, or don't apply to stdout or whatever source k8 uses for its logs? Given that the pods don't seem to have logs written to file anywhere, contrary to the properties, I'm inclined to say it's the former and that the log4j properties just aren't being picked up. Still have no idea why though.
On Tue, Dec 10, 2019 at 6:56 PM Yun Tang <myas...@live.com> wrote: > Sure, /opt/flink/conf is mounted as a volume from the configmap. > > > > Best > > Yun Tang > > > > *From: *Li Peng <li.p...@doordash.com> > *Date: *Wednesday, December 11, 2019 at 9:37 AM > *To: *Yang Wang <danrtsey...@gmail.com> > *Cc: *vino yang <yanghua1...@gmail.com>, user <user@flink.apache.org> > *Subject: *Re: Flink on Kubernetes seems to ignore log4j.properties > > > > 1. Hey Yun, I'm calling /opt/flink/bin/standalone-job.sh and > /opt/flink/bin/taskmanager.sh on my job and task managers respectively. > It's based on the setup described here: > http://shzhangji.com/blog/2019/08/24/deploy-flink-job-cluster-on-kubernetes/ . > I haven't tried the configmap approach yet, does it also replace the conf > files in /opt/flink/conf? > > 2. Hey Vino, here's a sample of the kubernetes: > https://pastebin.com/fqJrgjZu I didn't change any patterns from the > default, so the string patterns should look the same, but as you can see > it's full of info checkpoint logs that I originally was trying to suppress. > Based on my log4j.properties, the level should be set to WARN. I couldn't > actually find any .out files on the pod, this is from the kubectl logs > command. I also didn't see any files in /opt/flink/log, which I thought my > log4j was specified to do, hence me thinking that the properties weren't > actually being consumed. I also have the same properties in my > src/main/resources folder. > > 3. Hey Yang, yes this is a standalone session cluster. I did specify in > the docker file to copy the log4j.properties to the /opt/flink/conf folder > on the image, and I confirmed that the properties are correct when I bash'd > into the pod and viewed them manually. > > > > Incidentally, I also tried passing the -Dlog4j.configuration argument to > the programs, and it doesn't work either. And based on what I'm reading on > jira, that option is not really supported anymore? > > > > Thanks for your responses, folks! > > Li > > > > On Mon, Dec 9, 2019 at 7:10 PM Yang Wang <danrtsey...@gmail.com> wrote: > > Hi Li Peng, > > > > You are running standalone session cluster or per-job cluster on > kubernetes. Right? > > If so, i think you need to check your log4j.properties in the image, not > local. The log is > > stored to /opt/flink/log/jobmanager.log by default. > > > > If you are running active Kubernetes integration for a fresh taste. The > following cli option > > could be used to remove the redirect. > > -Dkubernetes.container-start-command-template="%java% %classpath% %jvmmem% > %jvmopts% %logging% %class% %args%" > > > > Best, > > Yang > > > > vino yang <yanghua1...@gmail.com> 于2019年12月10日周二 上午10:55写道： > > Hi Li, > > > > A potential reason could be conflicting logging frameworks. Can you share > the log in your .out file and let us know if the print format of the log is > the same as the configuration file you gave. > > > > Best, > > Vino > > > > Li Peng <li.p...@doordash.com> 于2019年12月10日周二 上午10:09写道： > > Hey folks, I noticed that my kubernetes flink logs (reached via *kubectl > logs <pod-name>*) completely ignore any of the configurations I put into > /flink/conf/. I set the logger level to WARN, yet I still see INFO level > logging from flink loggers > like org.apache.flink.runtime.checkpoint.CheckpointCoordinator. I even made > copied the same properties to /flink/conf/log4j-console.properties > and log4j-cli.properties. > > > > From what I can tell, kubernetes just listens to stdout and stderr, so > shouldn't the log4j.properties control output to them? Anyone seen this > issue before? > > > > Here is my log4j.properties: > > > # This affects logging for both user code and Flink > log4j.rootLogger=WARN, file, console, stdout > > # Uncomment this if you want to _only_ change Flink's logging > log4j.logger.org.apache.flink=WARN > > # The following lines keep the log level of common libraries/connectors on > # log level INFO. The root logger does not override this. You have to manually > # change the log levels here. > log4j.logger.akka=INFO > log4j.logger.org.apache.kafka=INFO > log4j.logger.org.apache.hadoop=INFO > log4j.logger.org.apache.zookeeper=INFO > > # Log all infos in the given file > log4j.appender.file=org.apache.log4j.FileAppender > log4j.appender.file.file=${log.file} > log4j.appender.file.append=false > log4j.appender.file.layout=org.apache.log4j.PatternLayout > log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p > %-60c %x - %m%n > > # Log all infos to the console > log4j.appender.console=org.apache.log4j.ConsoleAppender > log4j.appender.console.layout=org.apache.log4j.PatternLayout > log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} > %-5p %-60c %x - %m%n > > # Suppress the irrelevant (wrong) warnings from the Netty channel handler > log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, > file, console > log4j.logger.org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction=WARN > log4j.logger.org.apache.flink.runtime.checkpoint=WARN > > Thanks, > > Li > >
