Good to know - very helpful. Thank you both, Joe W. and Mark P. On Thu, Feb 15, 2024 at 10:42 AM Joe Witt <joe.w...@gmail.com> wrote:
> This [1] blog seems amazingly appropriate and wow do we need these/any > such fields we intend to truly honor in a prominent place in the docs. > Super useful... > > [1] https://jameswing.net/nifi/nifi-internal-fields.html > > Thanks > > On Thu, Feb 15, 2024 at 8:35 AM Mark Payne <marka...@hotmail.com> wrote: > >> Jim, >> >> You can actually reference “lastQueueDate” in Expression Language. It is >> formatted as number of milliseconds since epoch. >> >> So you might have a RouteOnAttribute that has a property named “old” with >> a value of: >> ${lastQueueDate:lt( ${now():minus(10000)} )} >> >> So any FlowFile that has been queued for more than 10 seconds would be >> routed to “old”, anything else to “unmatched” >> >> Thanks >> -Mark >> >> >> On Feb 15, 2024, at 10:18 AM, James McMahon <jsmcmah...@gmail.com> wrote: >> >> That would work - what a good suggestion. I'll do that. I can format the >> resulting number and then RouteOnAttribute by the desired subset of the >> result. >> Something like this to set attribute dt.failure: >> >> ${now():toNumber():toDate("yyyy-MM-ddHH:mm:ss"):format("yyyyMMddHHmmss","EST")} >> Then I can effectively route the files. >> Thank you Jim S. >> >> On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey <jrsteineb...@gmail.com> >> wrote: >> >>> You could add an UpdateAttribute processor first in the failure path to >>> add a new attribute which contains the time the error occurred by using the >>> ${now()} or ${now():toNumber()} expression language function. >>> >>> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now >>> >>> Then later on in the flow you can compare current time to the saved >>> error time to see how much time has elapsed. >>> >>> — Jim >>> >>> >>> On Feb 15, 2024, at 9:44 AM, James McMahon <jsmcmah...@gmail.com> wrote: >>> >>> As it turns out lineageStartDate and Queued Duration are very different. >>> Without being able to get at Queued Duration as an attribute, it appears we >>> cannot RouteOnAttribute to filter thousands in a queue by anything like >>> hours they have been in queue. >>> Why would this be helpful? Let us say we have an InvokeHttp processor >>> making calls to a REST endpoint. We leave for a weekend and return to find >>> 5000 files in the Failure queue from this processor. It would be most >>> helpful to identify the start time and end time of these 5000 failures. We >>> can't do that reviewing only the first 100 flowfiles in the queue from the >>> UI. >>> One can make an assumption that all of these 5000 flowfiles that failed >>> InvokeHttp share a similar range of lineageStartDate, but that will not >>> necessarily be true depending on flow complexity. >>> >>> On Wed, Feb 14, 2024 at 9:49 AM James McMahon <jsmcmah...@gmail.com> >>> wrote: >>> >>>> What a great workaround, thank you once again Mike. I'll put this in >>>> and use it now. >>>> Jim >>>> >>>> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser <moser...@gmail.com> >>>> wrote: >>>> >>>>> Hello James, >>>>> >>>>> I'm not aware of a way to access Queued Duration using expression >>>>> language, but you can access the Lineage Duration information. The >>>>> Getting >>>>> Started Guide mentions both entryDate and lineageStartDate as immutable >>>>> attributes on all flowfiles. These are numbers of milliseconds since >>>>> epoch. If you need them in a readable format, you can use the format() >>>>> function. >>>>> >>>>> simple examples: >>>>> ${entryDate} = 1707859943778 >>>>> ${lineageStartDate} = 1707859943778 >>>>> ${lineageStartDate:format("yyyy-MM-dd HH:mm:ss.SSS")} = 2024-02-13 >>>>> 21:32:23.778 >>>>> >>>>> -- Mike >>>>> >>>>> >>>>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon <jsmcmah...@gmail.com> >>>>> wrote: >>>>> >>>>>> When we examine the contents of a queue through the UI and select a >>>>>> flowfile from the resulting list, we see FlowFile Details in the Details >>>>>> tab. Are those key/values accessible from nifi expression language? I >>>>>> would >>>>>> like to access Queued Duration. I have a queue that holds flowfiles with >>>>>> non-successful return codes for calls to REST services, and I want to >>>>>> route >>>>>> depending on how long these flowfiles have been sitting in my error queue >>>>>> to isolate the window when the REST service was unavailable. >>>>>> Thank you for any examples that show how we can access these keys and >>>>>> values. >>>>>> >>>>> >>> >>
