This [1] blog seems amazingly appropriate and wow do we need these/any such fields we intend to truly honor in a prominent place in the docs. Super useful...
[1] https://jameswing.net/nifi/nifi-internal-fields.html Thanks On Thu, Feb 15, 2024 at 8:35 AM Mark Payne <marka...@hotmail.com> wrote: > Jim, > > You can actually reference “lastQueueDate” in Expression Language. It is > formatted as number of milliseconds since epoch. > > So you might have a RouteOnAttribute that has a property named “old” with > a value of: > ${lastQueueDate:lt( ${now():minus(10000)} )} > > So any FlowFile that has been queued for more than 10 seconds would be > routed to “old”, anything else to “unmatched” > > Thanks > -Mark > > > On Feb 15, 2024, at 10:18 AM, James McMahon <jsmcmah...@gmail.com> wrote: > > That would work - what a good suggestion. I'll do that. I can format the > resulting number and then RouteOnAttribute by the desired subset of the > result. > Something like this to set attribute dt.failure: > > ${now():toNumber():toDate("yyyy-MM-ddHH:mm:ss"):format("yyyyMMddHHmmss","EST")} > Then I can effectively route the files. > Thank you Jim S. > > On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey <jrsteineb...@gmail.com> > wrote: > >> You could add an UpdateAttribute processor first in the failure path to >> add a new attribute which contains the time the error occurred by using the >> ${now()} or ${now():toNumber()} expression language function. >> >> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now >> >> Then later on in the flow you can compare current time to the saved error >> time to see how much time has elapsed. >> >> — Jim >> >> >> On Feb 15, 2024, at 9:44 AM, James McMahon <jsmcmah...@gmail.com> wrote: >> >> As it turns out lineageStartDate and Queued Duration are very different. >> Without being able to get at Queued Duration as an attribute, it appears we >> cannot RouteOnAttribute to filter thousands in a queue by anything like >> hours they have been in queue. >> Why would this be helpful? Let us say we have an InvokeHttp processor >> making calls to a REST endpoint. We leave for a weekend and return to find >> 5000 files in the Failure queue from this processor. It would be most >> helpful to identify the start time and end time of these 5000 failures. We >> can't do that reviewing only the first 100 flowfiles in the queue from the >> UI. >> One can make an assumption that all of these 5000 flowfiles that failed >> InvokeHttp share a similar range of lineageStartDate, but that will not >> necessarily be true depending on flow complexity. >> >> On Wed, Feb 14, 2024 at 9:49 AM James McMahon <jsmcmah...@gmail.com> >> wrote: >> >>> What a great workaround, thank you once again Mike. I'll put this in and >>> use it now. >>> Jim >>> >>> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser <moser...@gmail.com> >>> wrote: >>> >>>> Hello James, >>>> >>>> I'm not aware of a way to access Queued Duration using expression >>>> language, but you can access the Lineage Duration information. The Getting >>>> Started Guide mentions both entryDate and lineageStartDate as immutable >>>> attributes on all flowfiles. These are numbers of milliseconds since >>>> epoch. If you need them in a readable format, you can use the format() >>>> function. >>>> >>>> simple examples: >>>> ${entryDate} = 1707859943778 >>>> ${lineageStartDate} = 1707859943778 >>>> ${lineageStartDate:format("yyyy-MM-dd HH:mm:ss.SSS")} = 2024-02-13 >>>> 21:32:23.778 >>>> >>>> -- Mike >>>> >>>> >>>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon <jsmcmah...@gmail.com> >>>> wrote: >>>> >>>>> When we examine the contents of a queue through the UI and select a >>>>> flowfile from the resulting list, we see FlowFile Details in the Details >>>>> tab. Are those key/values accessible from nifi expression language? I >>>>> would >>>>> like to access Queued Duration. I have a queue that holds flowfiles with >>>>> non-successful return codes for calls to REST services, and I want to >>>>> route >>>>> depending on how long these flowfiles have been sitting in my error queue >>>>> to isolate the window when the REST service was unavailable. >>>>> Thank you for any examples that show how we can access these keys and >>>>> values. >>>>> >>>> >> >
