Is result milliseconds since epoch?

[James McMahon]

I have a string representation of a datetime parsed from a lengthy
filename, attribute myValue with value of 20221003055959. If I convert to a
true datetime value and then apply toNumber() to that, does toNumber()
return milliseconds since the epoch? The expression language guide doesn’t
go into too much detail.
Thank you.

Re: Trouble configuring logging

[Kevin Doran]

 Thanks Dylan - Thanks for your interest in contributing to a solution for
the issue! Let's collaborate on the Jira.

Cheers,
Kevin

On Oct 3, 2022 at 12:59:02, Dylan Klomparens 
wrote:

> Pardon me, I forgot to include the link to the JIRA ticket:
> https://issues.apache.org/jira/browse/NIFI-10579
> --
> *From:* Dylan Klomparens 
> *Sent:* Monday, October 3, 2022 12:16 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Trouble configuring logging
>
> Everyone, thank you for the feedback on this topic.
>
> I have created a JIRA Improvement ticket about this topic. I proposed that
> there be a way to disable redirection of standard out and standard error to
> logback.
>
> If there is a positive response from the community and Apache foundation
> members, I will implement this improvement - following the contribution
> guide 
> - and submit a pull request.
>
>
> --
> *From:* Chris Sampson 
> *Sent:* Thursday, September 29, 2022 6:52 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Trouble configuring logging
>
> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
> In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran  wrote:
>
> Yeah, if it helps at all, this is how we work around this in the Apache
> Dockerfile (I knew realize why this method was used rather than a standard
> console appender in logback):
>
> # Continuously provide logs so that 'docker logs' can produce them
> "${NIFI_HOME}/bin/nifi.sh" run &
> nifi_pid="$!"
> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>
>
> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>
>
> Something to look into. As more NiFi deployments become containerized,
> this may be an area for improvement over time.
>
> Thanks,
> Kevin
>
> On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
> wrote:
>
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> 
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> --
> *From:* Mark Payne 
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users 
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it 

Re: Trouble configuring logging

[Dylan Klomparens]

Pardon me, I forgot to include the link to the JIRA ticket:
https://issues.apache.org/jira/browse/NIFI-10579

From: Dylan Klomparens 
Sent: Monday, October 3, 2022 12:16 PM
To: users@nifi.apache.org 
Subject: Re: Trouble configuring logging

Everyone, thank you for the feedback on this topic.

I have created a JIRA Improvement ticket about this topic. I proposed that 
there be a way to disable redirection of standard out and standard error to 
logback.

If there is a positive response from the community and Apache foundation 
members, I will implement this improvement - following the contribution 
guide - and 
submit a pull request.



From: Chris Sampson 
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org 
Subject: Re: Trouble configuring logging

We had similar logging configuration issues when using the apache/nifi image in 
a Kubernetes environment, so had to adjust how we log things via the 
logback.xml to make formats a little more consistent, remove some logs we 
didn't (generally) find so useful and hide some things in log files within the 
container because it would cause policy/GDPR issues if it was output to StdOut 
where it could potentially be exposed to other tenants within our multi-tenant 
cloud.

A cutdown/sanitised version of this logback.xml file has been uploaded as a 
gist [1] in case it helps anyone in future - pay attention to the comments 
within the XML if you want to use the file as it's not complete (I've removed 
some duplicated appender definitions where the only difference was the appender 
name and output log file, for example).

We also customised the image with a different startup script that is 
effectively the same as the apache/nifi start.sh but doesn't include the "tail" 
of the "nifi-app.log" file (as we no longer have that file in our config and 
the logback.xml sends everything to StdOut anyway).

In terms of a future approach, I wonder whether taking a similar approach to 
some other applications, like Elasticsearch, would be sensible where they 
include different default configurations in the various distribution mechanisms 
they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more 
things configurable via environment variables would also be worth considering 
if someone has time because Spring Boot/Logback can definitely handle that, 
allowing users to then configure their deployment with little more than 
environment variables, etc.


[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811

---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com


On Thu, 29 Sept 2022 at 01:21, Kevin Doran 
mailto:kdo...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache 
Dockerfile (I knew realize why this method was used rather than a standard 
console appender in logback):

# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &

https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129

Something to look into. As more NiFi deployments become containerized, this may 
be an area for improvement over time.

Thanks,
Kevin

On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information 
allowed me to piece together the puzzle. Indeed, anything that is sent to 
standard out subsequently causes the log message to come from the the 
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me 
to find the exact line of code that does 
this.

From my perspective this is an unfortunate design choice for NiFi because it 
does not allow standard out to be redirected easily. (For example, flowing 
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log 
driver, in my case). I suppose I'll have to find an alternative logback 
appender class to output the logs to, then find a way to forward them on from 
there.

Thanks again for the additional information that put the picture into view.

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users mailto:users@nifi.apache.org>>
Subject: Re: Trouble configuring logging


[EXTERNAL]

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) 
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi 
process is responsible for doing a lot of things, including monitoring the nifi 
process and if 

Re: Trouble configuring logging

[Dylan Klomparens]

Everyone, thank you for the feedback on this topic.

I have created a JIRA Improvement ticket about this topic. I proposed that 
there be a way to disable redirection of standard out and standard error to 
logback.

If there is a positive response from the community and Apache foundation 
members, I will implement this improvement - following the contribution 
guide - and 
submit a pull request.



From: Chris Sampson 
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org 
Subject: Re: Trouble configuring logging

We had similar logging configuration issues when using the apache/nifi image in 
a Kubernetes environment, so had to adjust how we log things via the 
logback.xml to make formats a little more consistent, remove some logs we 
didn't (generally) find so useful and hide some things in log files within the 
container because it would cause policy/GDPR issues if it was output to StdOut 
where it could potentially be exposed to other tenants within our multi-tenant 
cloud.

A cutdown/sanitised version of this logback.xml file has been uploaded as a 
gist [1] in case it helps anyone in future - pay attention to the comments 
within the XML if you want to use the file as it's not complete (I've removed 
some duplicated appender definitions where the only difference was the appender 
name and output log file, for example).

We also customised the image with a different startup script that is 
effectively the same as the apache/nifi start.sh but doesn't include the "tail" 
of the "nifi-app.log" file (as we no longer have that file in our config and 
the logback.xml sends everything to StdOut anyway).

In terms of a future approach, I wonder whether taking a similar approach to 
some other applications, like Elasticsearch, would be sensible where they 
include different default configurations in the various distribution mechanisms 
they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more 
things configurable via environment variables would also be worth considering 
if someone has time because Spring Boot/Logback can definitely handle that, 
allowing users to then configure their deployment with little more than 
environment variables, etc.


[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811

---
Chris Sampson
IT Consultant
chris.samp...@naimuri.com


On Thu, 29 Sept 2022 at 01:21, Kevin Doran 
mailto:kdo...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache 
Dockerfile (I knew realize why this method was used rather than a standard 
console appender in logback):

# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &

https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129

Something to look into. As more NiFi deployments become containerized, this may 
be an area for improvement over time.

Thanks,
Kevin

On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information 
allowed me to piece together the puzzle. Indeed, anything that is sent to 
standard out subsequently causes the log message to come from the the 
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me 
to find the exact line of code that does 
this.

From my perspective this is an unfortunate design choice for NiFi because it 
does not allow standard out to be redirected easily. (For example, flowing 
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log 
driver, in my case). I suppose I'll have to find an alternative logback 
appender class to output the logs to, then find a way to forward them on from 
there.

Thanks again for the additional information that put the picture into view.

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users mailto:users@nifi.apache.org>>
Subject: Re: Trouble configuring logging


[EXTERNAL]

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) 
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi 
process is responsible for doing a lot of things, including monitoring the nifi 
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs 
it.
So you’d probably need to update the logging for the appender writing to the 
bootstrap file:
mailto:kdo...@apache.org>> wrote:

Dylan - I looked into this and