Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there. Thanks again for the additional information that put the picture into view. ________________________________ From: Mark Payne <marka...@hotmail.com> Sent: Wednesday, September 28, 2022 10:14 AM To: users <users@nifi.apache.org> Subject: Re: Trouble configuring logging [EXTERNAL] This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it. Anything written to NiFi’s Standard Out goes to this processor, which then logs it. So you’d probably need to update the logging for the appender writing to the bootstrap file: <appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender”> And redirect that to standard out Thanks -Mark On Sep 28, 2022, at 9:48 AM, Kevin Doran <kdo...@apache.org<mailto:kdo...@apache.org>> wrote: Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile? Thanks, Kevin On Sep 27, 2022 at 10:21:12, Dylan Klomparens <dklompar...@foodallergy.org<mailto:dklompar...@foodallergy.org>> wrote: I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console. The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them. It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message: 2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from? My logback.xml configuration is below for reference. <?xml version="1.0" encoding="UTF-8"?> <configuration debug="true"> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"> <resetJUL>true</resetJUL> </contextListener> <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <logger name="org.apache.nifi" level="INFO"/> <logger name="org.apache.nifi.processors" level="WARN"/> <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/> <logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/> <logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" /> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" /> <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" level="ERROR" /> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> <logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" /> <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" level="ERROR" /> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" /> <logger name="org.apache.curator.ConnectionState" level="OFF" /> <!-- Logger for managing logging statements for nifi clusters. --> <logger name="org.apache.nifi.cluster" level="INFO"/> <!-- Logger for logging HTTP requests received by the web server. --> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> <!-- Logger for managing logging statements for jetty --> <logger name="org.eclipse.jetty" level="INFO"/> <!-- Suppress non-error messages due to excessive logging by class or library --> <logger name="org.springframework" level="ERROR"/> <!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) --> <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/> <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. --> <logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/> <!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default --> <logger name="net.schmizz.sshj" level="WARN" /> <logger name="com.hierynomus.sshj" level="WARN" /> <!-- Suppress non-error messages from SMBJ which was emitting large amounts of INFO logs by default --> <logger name="com.hierynomus.smbj" level="WARN" /> <!-- Suppress non-error messages from AWS KCL which was emitting large amounts of INFO logs by default --> <logger name="com.amazonaws.services.kinesis" level="WARN" /> <!-- Suppress non-error messages from Apache Atlas which was emitting large amounts of INFO logs by default --> <logger name="org.apache.atlas" level="WARN" /> <logger name="org.apache.nifi.web.security.requests" level="INFO" /> <!-- Logger for capturing user events. We do not want to propagate these log events to the root logger. These messages are only sent to the user-log appender. --> <logger name="org.apache.nifi.web.security" level="INFO" /> <logger name="org.apache.nifi.web.api.config" level="INFO" /> <logger name="org.apache.nifi.authorization" level="INFO" /> <logger name="org.apache.nifi.cluster.authorization" level="INFO" /> <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" /> <logger name="org.springframework.security.saml.log" level="WARN" /> <logger name="org.opensaml" level="WARN" /> <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" /> <logger name="org.apache.nifi.bootstrap" level="INFO" /> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" /> <logger name="org.apache.nifi.StdOut" level="INFO" /> <logger name="org.apache.nifi.StdErr" level="INFO" /> <root level="INFO"> <appender-ref ref="CONSOLE" /> </root> </configuration>