Re: Trouble configuring logging
Thanks Dylan - Thanks for your interest in contributing to a solution for the issue! Let's collaborate on the Jira. Cheers, Kevin On Oct 3, 2022 at 12:59:02, Dylan Klomparens wrote: > Pardon me, I forgot to include the link to the JIRA ticket: > https://issues.apache.org/jira/browse/NIFI-10579 > -- > *From:* Dylan Klomparens > *Sent:* Monday, October 3, 2022 12:16 PM > *To:* users@nifi.apache.org > *Subject:* Re: Trouble configuring logging > > Everyone, thank you for the feedback on this topic. > > I have created a JIRA Improvement ticket about this topic. I proposed that > there be a way to disable redirection of standard out and standard error to > logback. > > If there is a positive response from the community and Apache foundation > members, I will implement this improvement - following the contribution > guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> > - and submit a pull request. > > > -- > *From:* Chris Sampson > *Sent:* Thursday, September 29, 2022 6:52 AM > *To:* users@nifi.apache.org > *Subject:* Re: Trouble configuring logging > > We had similar logging configuration issues when using the apache/nifi > image in a Kubernetes environment, so had to adjust how we log things via > the logback.xml to make formats a little more consistent, remove some logs > we didn't (generally) find so useful and hide some things in log files > within the container because it would cause policy/GDPR issues if it was > output to StdOut where it could potentially be exposed to other tenants > within our multi-tenant cloud. > > A cutdown/sanitised version of this logback.xml file has been uploaded as > a gist [1] in case it helps anyone in future - pay attention to the > comments within the XML if you want to use the file as it's not complete > (I've removed some duplicated appender definitions where the only > difference was the appender name and output log file, for example). > > We also customised the image with a different startup script that is > effectively the same as the apache/nifi start.sh but doesn't include the > "tail" of the "nifi-app.log" file (as we no longer have that file in our > config and the logback.xml sends everything to StdOut anyway). > > In terms of a future approach, I wonder whether taking a similar approach > to some other applications, like Elasticsearch, would be sensible where > they include different default configurations in the various distribution > mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the > work to make more things configurable via environment variables would also > be worth considering if someone has time because Spring Boot/Logback can > definitely handle that, allowing users to then configure their deployment > with little more than environment variables, etc. > > > [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 > > --- > *Chris Sampson* > IT Consultant > chris.samp...@naimuri.com > > > On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote: > > Yeah, if it helps at all, this is how we work around this in the Apache > Dockerfile (I knew realize why this method was used rather than a standard > console appender in logback): > > # Continuously provide logs so that 'docker logs' can produce them > "${NIFI_HOME}/bin/nifi.sh" run & > nifi_pid="$!" > tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & > > > https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 > > > Something to look into. As more NiFi deployments become containerized, > this may be an area for improvement over time. > > Thanks, > Kevin > > On Sep 28, 2022 at 19:40:35, Dylan Klomparens > wrote: > > Mark and Kevin, thank you for your insightful comments. That information > allowed me to piece together the puzzle. Indeed, anything that is sent to > standard out subsequently *causes* the log message to come from the the > org.apache.nifi.StdOut logger in the RunNiFi class. Your description > allowed me to find the exact line of code that does this > <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> > . > > From my perspective this is an unfortunate design choice for NiFi because > it does not allow standard out to be redirected easily. (For example, > flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's > built-in log driver, in my case). I suppose I'll have to find an > alternative logback appender class to output the logs to, then find a way >
Re: Trouble configuring logging
Pardon me, I forgot to include the link to the JIRA ticket: https://issues.apache.org/jira/browse/NIFI-10579 From: Dylan Klomparens Sent: Monday, October 3, 2022 12:16 PM To: users@nifi.apache.org Subject: Re: Trouble configuring logging Everyone, thank you for the feedback on this topic. I have created a JIRA Improvement ticket about this topic. I proposed that there be a way to disable redirection of standard out and standard error to logback. If there is a positive response from the community and Apache foundation members, I will implement this improvement - following the contribution guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and submit a pull request. From: Chris Sampson Sent: Thursday, September 29, 2022 6:52 AM To: users@nifi.apache.org Subject: Re: Trouble configuring logging We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud. A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example). We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway). In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc. [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 --- Chris Sampson IT Consultant chris.samp...@naimuri.com<mailto:chris.samp...@naimuri.com> On Thu, 29 Sept 2022 at 01:21, Kevin Doran mailto:kdo...@apache.org>> wrote: Yeah, if it helps at all, this is how we work around this in the Apache Dockerfile (I knew realize why this method was used rather than a standard console appender in logback): # Continuously provide logs so that 'docker logs' can produce them "${NIFI_HOME}/bin/nifi.sh" run & nifi_pid="$!" tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 Something to look into. As more NiFi deployments become containerized, this may be an area for improvement over time. Thanks, Kevin On Sep 28, 2022 at 19:40:35, Dylan Klomparens mailto:dklompar...@foodallergy.org>> wrote: Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>. From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there. Thanks again for the additional information that put the picture into view. From: Mark Payne mailto:marka...@hotmail.com>> Sent: Wednesday, September 28, 2022 10:14 AM To: users mailto:users@nifi.apache.org>> Subject: Re: Trouble configuring logging [EXTERNAL] This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. Th
Re: Trouble configuring logging
Everyone, thank you for the feedback on this topic. I have created a JIRA Improvement ticket about this topic. I proposed that there be a way to disable redirection of standard out and standard error to logback. If there is a positive response from the community and Apache foundation members, I will implement this improvement - following the contribution guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and submit a pull request. From: Chris Sampson Sent: Thursday, September 29, 2022 6:52 AM To: users@nifi.apache.org Subject: Re: Trouble configuring logging We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud. A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example). We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway). In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc. [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 --- Chris Sampson IT Consultant chris.samp...@naimuri.com<mailto:chris.samp...@naimuri.com> On Thu, 29 Sept 2022 at 01:21, Kevin Doran mailto:kdo...@apache.org>> wrote: Yeah, if it helps at all, this is how we work around this in the Apache Dockerfile (I knew realize why this method was used rather than a standard console appender in logback): # Continuously provide logs so that 'docker logs' can produce them "${NIFI_HOME}/bin/nifi.sh" run & nifi_pid="$!" tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 Something to look into. As more NiFi deployments become containerized, this may be an area for improvement over time. Thanks, Kevin On Sep 28, 2022 at 19:40:35, Dylan Klomparens mailto:dklompar...@foodallergy.org>> wrote: Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>. From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there. Thanks again for the additional information that put the picture into view. From: Mark Payne mailto:marka...@hotmail.com>> Sent: Wednesday, September 28, 2022 10:14 AM To: users mailto:users@nifi.apache.org>> Subject: Re: Trouble configuring logging [EXTERNAL] This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it. Anything written to NiFi’s Standard Out goes to this processor, which then logs it. So you’d probably need to update the logging for
Re: Trouble configuring logging
Thanks Chris! This looks very helpful. In terms of a future approach, I wonder whether taking a similar approach > to some other applications, like Elasticsearch, would be sensible where > they include different default configurations in the various distribution > mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the > work to make more things configurable via environment variables would also > be worth considering if someone has time because Spring Boot/Logback can > definitely handle that, allowing users to then configure their deployment > with little more than environment variables, etc. > Yeah I love that idea. I linked this email thread on our Docker Improvements page [1], where we have a placeholder for logging improvements. I actually may have time to make some contributions to the Docker Hub image in coming months 爛 [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements Cheers, Kevin On Sep 29, 2022 at 06:52:57, Chris Sampson wrote: > We had similar logging configuration issues when using the apache/nifi > image in a Kubernetes environment, so had to adjust how we log things via > the logback.xml to make formats a little more consistent, remove some logs > we didn't (generally) find so useful and hide some things in log files > within the container because it would cause policy/GDPR issues if it was > output to StdOut where it could potentially be exposed to other tenants > within our multi-tenant cloud. > > A cutdown/sanitised version of this logback.xml file has been uploaded as > a gist [1] in case it helps anyone in future - pay attention to the > comments within the XML if you want to use the file as it's not complete > (I've removed some duplicated appender definitions where the only > difference was the appender name and output log file, for example). > > We also customised the image with a different startup script that is > effectively the same as the apache/nifi start.sh but doesn't include the > "tail" of the "nifi-app.log" file (as we no longer have that file in our > config and the logback.xml sends everything to StdOut anyway). > > > > > [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 > > --- > *Chris Sampson* > IT Consultant > chris.samp...@naimuri.com > > > On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote: > >> Yeah, if it helps at all, this is how we work around this in the Apache >> Dockerfile (I knew realize why this method was used rather than a standard >> console appender in logback): >> >> # Continuously provide logs so that 'docker logs' can produce them >> "${NIFI_HOME}/bin/nifi.sh" run & >> nifi_pid="$!" >> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & >> >> >> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 >> >> >> Something to look into. As more NiFi deployments become containerized, >> this may be an area for improvement over time. >> >> Thanks, >> Kevin >> >> On Sep 28, 2022 at 19:40:35, Dylan Klomparens < >> dklompar...@foodallergy.org> wrote: >> >>> Mark and Kevin, thank you for your insightful comments. That information >>> allowed me to piece together the puzzle. Indeed, anything that is sent to >>> standard out subsequently *causes* the log message to come from the the >>> org.apache.nifi.StdOut logger in the RunNiFi class. Your description >>> allowed me to find the exact line of code that does this >>> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> >>> . >>> >>> From my perspective this is an unfortunate design choice for NiFi >>> because it does not allow standard out to be redirected easily. (For >>> example, flowing onward to Docker, and forwarded to AWS CloudWatch using >>> Docker's built-in log driver, in my case). I suppose I'll have to find an >>> alternative logback appender class to output the logs to, then find a way >>> to forward them on from there. >>> >>> Thanks again for the additional information that put the picture into >>> view. >>> -- >>> *From:* Mark Payne >>> *Sent:* Wednesday, September 28, 2022 10:14 AM >>> *To:* users >>> *Subject:* Re: Trouble configuring logging >>> >>> >>> [EXTERNAL] >>> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh >>> start) it doesn’t directly start the NiFi pro
Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud. A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example). We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway). In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc. [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811 --- *Chris Sampson* IT Consultant chris.samp...@naimuri.com On Thu, 29 Sept 2022 at 01:21, Kevin Doran wrote: > Yeah, if it helps at all, this is how we work around this in the Apache > Dockerfile (I knew realize why this method was used rather than a standard > console appender in logback): > > # Continuously provide logs so that 'docker logs' can produce them > "${NIFI_HOME}/bin/nifi.sh" run & > nifi_pid="$!" > tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & > > > https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 > > > Something to look into. As more NiFi deployments become containerized, > this may be an area for improvement over time. > > Thanks, > Kevin > > On Sep 28, 2022 at 19:40:35, Dylan Klomparens > wrote: > >> Mark and Kevin, thank you for your insightful comments. That information >> allowed me to piece together the puzzle. Indeed, anything that is sent to >> standard out subsequently *causes* the log message to come from the the >> org.apache.nifi.StdOut logger in the RunNiFi class. Your description >> allowed me to find the exact line of code that does this >> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> >> . >> >> From my perspective this is an unfortunate design choice for NiFi because >> it does not allow standard out to be redirected easily. (For example, >> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's >> built-in log driver, in my case). I suppose I'll have to find an >> alternative logback appender class to output the logs to, then find a way >> to forward them on from there. >> >> Thanks again for the additional information that put the picture into >> view. >> -- >> *From:* Mark Payne >> *Sent:* Wednesday, September 28, 2022 10:14 AM >> *To:* users >> *Subject:* Re: Trouble configuring logging >> >> >> [EXTERNAL] >> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh >> start) it doesn’t directly start the NiFi process. >> Instead, it starts a different processor, which is called RunNiFi. This >> RunNiFi process is responsible for doing a lot of things, including >> monitoring the nifi process and if it dies restarting it. >> Anything written to NiFi’s Standard Out goes to this processor, which >> then logs it. >> So you’d probably need to update the logging for the appender writing to >> the bootstrap file: >> > class="ch.qos.logback.core.rolling.RollingFileAppender”> >> >> And redirect that to standard out >> >> Thanks >> -Mark >> >> >> On Sep 28, 2022, at 9:48 AM, Kevin Doran wrote: >> >> Dylan - I looked into this and am yet unable to offer an explaination. >> Perhaps others that are familiar with how org.apache.nifi.StdOut
Re: Trouble configuring logging
Yeah, if it helps at all, this is how we work around this in the Apache Dockerfile (I knew realize why this method was used rather than a standard console appender in logback): # Continuously provide logs so that 'docker logs' can produce them "${NIFI_HOME}/bin/nifi.sh" run & nifi_pid="$!" tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" & https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129 Something to look into. As more NiFi deployments become containerized, this may be an area for improvement over time. Thanks, Kevin On Sep 28, 2022 at 19:40:35, Dylan Klomparens wrote: > Mark and Kevin, thank you for your insightful comments. That information > allowed me to piece together the puzzle. Indeed, anything that is sent to > standard out subsequently *causes* the log message to come from the the > org.apache.nifi.StdOut logger in the RunNiFi class. Your description > allowed me to find the exact line of code that does this > <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485> > . > > From my perspective this is an unfortunate design choice for NiFi because > it does not allow standard out to be redirected easily. (For example, > flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's > built-in log driver, in my case). I suppose I'll have to find an > alternative logback appender class to output the logs to, then find a way > to forward them on from there. > > Thanks again for the additional information that put the picture into view. > -- > *From:* Mark Payne > *Sent:* Wednesday, September 28, 2022 10:14 AM > *To:* users > *Subject:* Re: Trouble configuring logging > > > [EXTERNAL] > This is because of how NiFi is run. When you startup nifi (bin/nifi.sh > start) it doesn’t directly start the NiFi process. > Instead, it starts a different processor, which is called RunNiFi. This > RunNiFi process is responsible for doing a lot of things, including > monitoring the nifi process and if it dies restarting it. > Anything written to NiFi’s Standard Out goes to this processor, which then > logs it. > So you’d probably need to update the logging for the appender writing to > the bootstrap file: > class="ch.qos.logback.core.rolling.RollingFileAppender”> > > And redirect that to standard out > > Thanks > -Mark > > > On Sep 28, 2022, at 9:48 AM, Kevin Doran wrote: > > Dylan - I looked into this and am yet unable to offer an explaination. > Perhaps others that are familiar with how org.apache.nifi.StdOut can shed > some light, or else I will keep digging when I have a block of time. To > help in my understanding: Which Docker image are you using? Is it the > apace/nifi image or a custom one, and if custom, can you share the > Dockerfile? > > Thanks, > Kevin > > On Sep 27, 2022 at 10:21:12, Dylan Klomparens > wrote: > > I am attempting to configure logging for NiFi. I have NiFi running in a > Docker container, which sends all console logs to AWS CloudWatch. > Therefore, I am configuring NiFi to send all logs to the console. > > The problem is, for some reason *all log messages are coming from the > org.apache.nifi.StdOut logger*. I cannot figure out why, since I would > like messages to be printed directly from the logger that is receiving them. > > It seems like messages are "passing through" loggers, which are ultimately > printed out from the org.apache.nifi.StdOut logger. Here is an example of > *one* log message: > *2022-09-27 10:08:01,849 INFO [NiFi logging handler] > org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1] > o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile > Repository* > > *Why would every single log message come from the StdOut logger? And how > can I have logs delivered from the logger they're supposedly originally > coming from?* > > My logback.xml configuration is below for reference. > > > > > > > true > > > > > %date %level [%thread] %logger{40} %msg%n > > > > > > level="INFO"/> > level="INFO"/> > name="org.apache.nifi.controller.repository.StandardProcessSession" > level="WARN" /> > > > > level="ERROR" /> > level="ERROR" /> > > > level="ERROR" /> > level="ERROR" /> > > > > level="OFF" /> > > > > > > > > > > > > > > > > > > > level="ERROR"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Trouble configuring logging
Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>. From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there. Thanks again for the additional information that put the picture into view. From: Mark Payne Sent: Wednesday, September 28, 2022 10:14 AM To: users Subject: Re: Trouble configuring logging [EXTERNAL] This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it. Anything written to NiFi’s Standard Out goes to this processor, which then logs it. So you’d probably need to update the logging for the appender writing to the bootstrap file: mailto:kdo...@apache.org>> wrote: Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile? Thanks, Kevin On Sep 27, 2022 at 10:21:12, Dylan Klomparens mailto:dklompar...@foodallergy.org>> wrote: I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console. The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them. It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message: 2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from? My logback.xml configuration is below for reference. true %date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it. Anything written to NiFi’s Standard Out goes to this processor, which then logs it. So you’d probably need to update the logging for the appender writing to the bootstrap file: mailto:kdo...@apache.org>> wrote: Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile? Thanks, Kevin On Sep 27, 2022 at 10:21:12, Dylan Klomparens mailto:dklompar...@foodallergy.org>> wrote: I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console. The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them. It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message: 2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from? My logback.xml configuration is below for reference. true %date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
Kevin, thank you for taking a look. I am using a custom Dockerfile (and docker-compose.yaml) I wrote, pasted at the end of this email. I also ran NiFi natively, without Docker, on my desktop computer and observed identical results. I initially suspected Docker could somehow be affecting stdout somehow, but in testing I've not found any evidence of that. Dockerfile FROM openjdk:11 # These are environment variables that are not meant to be # configurable, they are essential to the operation of the container. ENV NIFI_HOME /opt/nifi/ ENV NIFI_LOG_DIR /persistent-storage/logs ENV NIFI_OVERRIDE_NIFIENV true WORKDIR /opt # Unpack NiFi and Toolkit COPY nifi.zip /opt COPY nifi-toolkit.zip /opt RUN unzip nifi.zip RUN unzip nifi-toolkit.zip RUN mv nifi-1.17.0 nifi RUN mv nifi-toolkit-1.17.0 nifi-toolkit # Clean out unused files RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz RUN rm --force /opt/nifi/bin/*.bat RUN rm --force /opt/nifi/conf/* RUN rm --force /opt/nifi-toolkit/*.bat RUN mkdir /persistent-storage WORKDIR /opt/nifi # Set configuration COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml state-management.xml logback.xml /opt/nifi/conf/ COPY initialize.sh /opt/nifi/bin/ COPY postgres-jdbc-driver.jar /opt COPY snowflake-jdbc-driver.jar /opt COPY ZIP_codes_to_states.csv /opt CMD ["/opt/nifi/bin/nifi.sh", "run"] EXPOSE 8000/tcp docker-compose.yaml services: nifi: container_name: nifi build: . image: nifi volumes: [/persistent-storage/:/persistent-storage/] ports: [8000:8000] environment: [INITIAL_ADMIN_IDENTITY:'*redacted*'] logging: driver: "awslogs" options: awslogs-region: "*redacted*" awslogs-group: "*redacted*" awslogs-stream: "All logs" From: Kevin Doran Sent: Wednesday, September 28, 2022 9:48 AM To: users@nifi.apache.org Subject: Re: Trouble configuring logging [EXTERNAL] Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile? Thanks, Kevin On Sep 27, 2022 at 10:21:12, Dylan Klomparens mailto:dklompar...@foodallergy.org>> wrote: I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console. The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them. It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message: 2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from? My logback.xml configuration is below for reference. true %date %level [%thread] %logger{40} %msg%n
Re: Trouble configuring logging
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile? Thanks, Kevin On Sep 27, 2022 at 10:21:12, Dylan Klomparens wrote: > I am attempting to configure logging for NiFi. I have NiFi running in a > Docker container, which sends all console logs to AWS CloudWatch. > Therefore, I am configuring NiFi to send all logs to the console. > > The problem is, for some reason *all log messages are coming from the > org.apache.nifi.StdOut logger*. I cannot figure out why, since I would > like messages to be printed directly from the logger that is receiving them. > > It seems like messages are "passing through" loggers, which are ultimately > printed out from the org.apache.nifi.StdOut logger. Here is an example of > *one* log message: > *2022-09-27 10:08:01,849 INFO [NiFi logging handler] > org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1] > o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile > Repository* > > *Why would every single log message come from the StdOut logger? And how > can I have logs delivered from the logger they're supposedly originally > coming from?* > > My logback.xml configuration is below for reference. > > > > > > > true > > > > > %date %level [%thread] %logger{40} %msg%n > > > > > > level="INFO"/> > level="INFO"/> > name="org.apache.nifi.controller.repository.StandardProcessSession" > level="WARN" /> > > > > level="ERROR" /> > level="ERROR" /> > > > level="ERROR" /> > level="ERROR" /> > > > > level="OFF" /> > > > > > > > > > > > > > > > > > > > level="ERROR"/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Trouble configuring logging
I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console. The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them. It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message: 2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from? My logback.xml configuration is below for reference. true %date %level [%thread] %logger{40} %msg%n