Re: Can we access Queued Duration as an attribute?
Good to know - very helpful. Thank you both, Joe W. and Mark P.
On Thu, Feb 15, 2024 at 10:42 AM Joe Witt wrote:
> This [1] blog seems amazingly appropriate and wow do we need these/any
> such fields we intend to truly honor in a prominent place in the docs.
> Super useful...
>
> [1] https://jameswing.net/nifi/nifi-internal-fields.html
>
> Thanks
>
> On Thu, Feb 15, 2024 at 8:35 AM Mark Payne wrote:
>
>> Jim,
>>
>> You can actually reference “lastQueueDate” in Expression Language. It is
>> formatted as number of milliseconds since epoch.
>>
>> So you might have a RouteOnAttribute that has a property named “old” with
>> a value of:
>> ${lastQueueDate:lt( ${now():minus(1)} )}
>>
>> So any FlowFile that has been queued for more than 10 seconds would be
>> routed to “old”, anything else to “unmatched”
>>
>> Thanks
>> -Mark
>>
>>
>> On Feb 15, 2024, at 10:18 AM, James McMahon wrote:
>>
>> That would work - what a good suggestion. I'll do that. I can format the
>> resulting number and then RouteOnAttribute by the desired subset of the
>> result.
>> Something like this to set attribute dt.failure:
>>
>> ${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
>> Then I can effectively route the files.
>> Thank you Jim S.
>>
>> On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey
>> wrote:
>>
>>> You could add an UpdateAttribute processor first in the failure path to
>>> add a new attribute which contains the time the error occurred by using the
>>> ${now()} or ${now():toNumber()} expression language function.
>>>
>>> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
>>>
>>> Then later on in the flow you can compare current time to the saved
>>> error time to see how much time has elapsed.
>>>
>>> — Jim
>>>
>>>
>>> On Feb 15, 2024, at 9:44 AM, James McMahon wrote:
>>>
>>> As it turns out lineageStartDate and Queued Duration are very different.
>>> Without being able to get at Queued Duration as an attribute, it appears we
>>> cannot RouteOnAttribute to filter thousands in a queue by anything like
>>> hours they have been in queue.
>>> Why would this be helpful? Let us say we have an InvokeHttp processor
>>> making calls to a REST endpoint. We leave for a weekend and return to find
>>> 5000 files in the Failure queue from this processor. It would be most
>>> helpful to identify the start time and end time of these 5000 failures. We
>>> can't do that reviewing only the first 100 flowfiles in the queue from the
>>> UI.
>>> One can make an assumption that all of these 5000 flowfiles that failed
>>> InvokeHttp share a similar range of lineageStartDate, but that will not
>>> necessarily be true depending on flow complexity.
>>>
>>> On Wed, Feb 14, 2024 at 9:49 AM James McMahon
>>> wrote:
>>>
What a great workaround, thank you once again Mike. I'll put this in
and use it now.
Jim
On Tue, Feb 13, 2024 at 4:41 PM Michael Moser
wrote:
> Hello James,
>
> I'm not aware of a way to access Queued Duration using expression
> language, but you can access the Lineage Duration information. The
> Getting
> Started Guide mentions both entryDate and lineageStartDate as immutable
> attributes on all flowfiles. These are numbers of milliseconds since
> epoch. If you need them in a readable format, you can use the format()
> function.
>
> simple examples:
> ${entryDate} = 1707859943778
> ${lineageStartDate} = 1707859943778
> ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
> 21:32:23.778
>
> -- Mike
>
>
> On Mon, Feb 12, 2024 at 11:38 AM James McMahon
> wrote:
>
>> When we examine the contents of a queue through the UI and select a
>> flowfile from the resulting list, we see FlowFile Details in the Details
>> tab. Are those key/values accessible from nifi expression language? I
>> would
>> like to access Queued Duration. I have a queue that holds flowfiles with
>> non-successful return codes for calls to REST services, and I want to
>> route
>> depending on how long these flowfiles have been sitting in my error queue
>> to isolate the window when the REST service was unavailable.
>> Thank you for any examples that show how we can access these keys and
>> values.
>>
>
>>>
>>
Re: Can we access Queued Duration as an attribute?
This [1] blog seems amazingly appropriate and wow do we need these/any such
fields we intend to truly honor in a prominent place in the docs. Super
useful...
[1] https://jameswing.net/nifi/nifi-internal-fields.html
Thanks
On Thu, Feb 15, 2024 at 8:35 AM Mark Payne wrote:
> Jim,
>
> You can actually reference “lastQueueDate” in Expression Language. It is
> formatted as number of milliseconds since epoch.
>
> So you might have a RouteOnAttribute that has a property named “old” with
> a value of:
> ${lastQueueDate:lt( ${now():minus(1)} )}
>
> So any FlowFile that has been queued for more than 10 seconds would be
> routed to “old”, anything else to “unmatched”
>
> Thanks
> -Mark
>
>
> On Feb 15, 2024, at 10:18 AM, James McMahon wrote:
>
> That would work - what a good suggestion. I'll do that. I can format the
> resulting number and then RouteOnAttribute by the desired subset of the
> result.
> Something like this to set attribute dt.failure:
>
> ${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
> Then I can effectively route the files.
> Thank you Jim S.
>
> On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey
> wrote:
>
>> You could add an UpdateAttribute processor first in the failure path to
>> add a new attribute which contains the time the error occurred by using the
>> ${now()} or ${now():toNumber()} expression language function.
>>
>> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
>>
>> Then later on in the flow you can compare current time to the saved error
>> time to see how much time has elapsed.
>>
>> — Jim
>>
>>
>> On Feb 15, 2024, at 9:44 AM, James McMahon wrote:
>>
>> As it turns out lineageStartDate and Queued Duration are very different.
>> Without being able to get at Queued Duration as an attribute, it appears we
>> cannot RouteOnAttribute to filter thousands in a queue by anything like
>> hours they have been in queue.
>> Why would this be helpful? Let us say we have an InvokeHttp processor
>> making calls to a REST endpoint. We leave for a weekend and return to find
>> 5000 files in the Failure queue from this processor. It would be most
>> helpful to identify the start time and end time of these 5000 failures. We
>> can't do that reviewing only the first 100 flowfiles in the queue from the
>> UI.
>> One can make an assumption that all of these 5000 flowfiles that failed
>> InvokeHttp share a similar range of lineageStartDate, but that will not
>> necessarily be true depending on flow complexity.
>>
>> On Wed, Feb 14, 2024 at 9:49 AM James McMahon
>> wrote:
>>
>>> What a great workaround, thank you once again Mike. I'll put this in and
>>> use it now.
>>> Jim
>>>
>>> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser
>>> wrote:
>>>
Hello James,
I'm not aware of a way to access Queued Duration using expression
language, but you can access the Lineage Duration information. The Getting
Started Guide mentions both entryDate and lineageStartDate as immutable
attributes on all flowfiles. These are numbers of milliseconds since
epoch. If you need them in a readable format, you can use the format()
function.
simple examples:
${entryDate} = 1707859943778
${lineageStartDate} = 1707859943778
${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
21:32:23.778
-- Mike
On Mon, Feb 12, 2024 at 11:38 AM James McMahon
wrote:
> When we examine the contents of a queue through the UI and select a
> flowfile from the resulting list, we see FlowFile Details in the Details
> tab. Are those key/values accessible from nifi expression language? I
> would
> like to access Queued Duration. I have a queue that holds flowfiles with
> non-successful return codes for calls to REST services, and I want to
> route
> depending on how long these flowfiles have been sitting in my error queue
> to isolate the window when the REST service was unavailable.
> Thank you for any examples that show how we can access these keys and
> values.
>
>>
>
Re: Can we access Queued Duration as an attribute?
Jim,
You can actually reference “lastQueueDate” in Expression Language. It is
formatted as number of milliseconds since epoch.
So you might have a RouteOnAttribute that has a property named “old” with a
value of:
${lastQueueDate:lt( ${now():minus(1)} )}
So any FlowFile that has been queued for more than 10 seconds would be routed
to “old”, anything else to “unmatched”
Thanks
-Mark
On Feb 15, 2024, at 10:18 AM, James McMahon wrote:
That would work - what a good suggestion. I'll do that. I can format the
resulting number and then RouteOnAttribute by the desired subset of the result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
Then I can effectively route the files.
Thank you Jim S.
On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey
mailto:jrsteineb...@gmail.com>> wrote:
You could add an UpdateAttribute processor first in the failure path to add a
new attribute which contains the time the error occurred by using the ${now()}
or ${now():toNumber()} expression language function.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
Then later on in the flow you can compare current time to the saved error time
to see how much time has elapsed.
— Jim
On Feb 15, 2024, at 9:44 AM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
As it turns out lineageStartDate and Queued Duration are very different.
Without being able to get at Queued Duration as an attribute, it appears we
cannot RouteOnAttribute to filter thousands in a queue by anything like hours
they have been in queue.
Why would this be helpful? Let us say we have an InvokeHttp processor making
calls to a REST endpoint. We leave for a weekend and return to find 5000 files
in the Failure queue from this processor. It would be most helpful to identify
the start time and end time of these 5000 failures. We can't do that reviewing
only the first 100 flowfiles in the queue from the UI.
One can make an assumption that all of these 5000 flowfiles that failed
InvokeHttp share a similar range of lineageStartDate, but that will not
necessarily be true depending on flow complexity.
On Wed, Feb 14, 2024 at 9:49 AM James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
What a great workaround, thank you once again Mike. I'll put this in and use it
now.
Jim
On Tue, Feb 13, 2024 at 4:41 PM Michael Moser
mailto:moser...@gmail.com>> wrote:
Hello James,
I'm not aware of a way to access Queued Duration using expression language, but
you can access the Lineage Duration information. The Getting Started Guide
mentions both entryDate and lineageStartDate as immutable attributes on all
flowfiles. These are numbers of milliseconds since epoch. If you need them in
a readable format, you can use the format() function.
simple examples:
${entryDate} = 1707859943778
${lineageStartDate} = 1707859943778
${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13 21:32:23.778
-- Mike
On Mon, Feb 12, 2024 at 11:38 AM James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
When we examine the contents of a queue through the UI and select a flowfile
from the resulting list, we see FlowFile Details in the Details tab. Are those
key/values accessible from nifi expression language? I would like to access
Queued Duration. I have a queue that holds flowfiles with non-successful return
codes for calls to REST services, and I want to route depending on how long
these flowfiles have been sitting in my error queue to isolate the window when
the REST service was unavailable.
Thank you for any examples that show how we can access these keys and values.
Re: Can we access Queued Duration as an attribute?
That would work - what a good suggestion. I'll do that. I can format the
resulting number and then RouteOnAttribute by the desired subset of the
result.
Something like this to set attribute dt.failure:
${now():toNumber():toDate("-MM-ddHH:mm:ss"):format("MMddHHmmss","EST")}
Then I can effectively route the files.
Thank you Jim S.
On Thu, Feb 15, 2024 at 9:55 AM Jim Steinebrey
wrote:
> You could add an UpdateAttribute processor first in the failure path to
> add a new attribute which contains the time the error occurred by using the
> ${now()} or ${now():toNumber()} expression language function.
>
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
>
> Then later on in the flow you can compare current time to the saved error
> time to see how much time has elapsed.
>
> — Jim
>
>
> On Feb 15, 2024, at 9:44 AM, James McMahon wrote:
>
> As it turns out lineageStartDate and Queued Duration are very different.
> Without being able to get at Queued Duration as an attribute, it appears we
> cannot RouteOnAttribute to filter thousands in a queue by anything like
> hours they have been in queue.
> Why would this be helpful? Let us say we have an InvokeHttp processor
> making calls to a REST endpoint. We leave for a weekend and return to find
> 5000 files in the Failure queue from this processor. It would be most
> helpful to identify the start time and end time of these 5000 failures. We
> can't do that reviewing only the first 100 flowfiles in the queue from the
> UI.
> One can make an assumption that all of these 5000 flowfiles that failed
> InvokeHttp share a similar range of lineageStartDate, but that will not
> necessarily be true depending on flow complexity.
>
> On Wed, Feb 14, 2024 at 9:49 AM James McMahon
> wrote:
>
>> What a great workaround, thank you once again Mike. I'll put this in and
>> use it now.
>> Jim
>>
>> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser wrote:
>>
>>> Hello James,
>>>
>>> I'm not aware of a way to access Queued Duration using expression
>>> language, but you can access the Lineage Duration information. The Getting
>>> Started Guide mentions both entryDate and lineageStartDate as immutable
>>> attributes on all flowfiles. These are numbers of milliseconds since
>>> epoch. If you need them in a readable format, you can use the format()
>>> function.
>>>
>>> simple examples:
>>> ${entryDate} = 1707859943778
>>> ${lineageStartDate} = 1707859943778
>>> ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
>>> 21:32:23.778
>>>
>>> -- Mike
>>>
>>>
>>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon
>>> wrote:
>>>
When we examine the contents of a queue through the UI and select a
flowfile from the resulting list, we see FlowFile Details in the Details
tab. Are those key/values accessible from nifi expression language? I would
like to access Queued Duration. I have a queue that holds flowfiles with
non-successful return codes for calls to REST services, and I want to route
depending on how long these flowfiles have been sitting in my error queue
to isolate the window when the REST service was unavailable.
Thank you for any examples that show how we can access these keys and
values.
>>>
>
Re: Can we access Queued Duration as an attribute?
You could add an UpdateAttribute processor first in the failure path to add a
new attribute which contains the time the error occurred by using the ${now()}
or ${now():toNumber()} expression language function.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now
Then later on in the flow you can compare current time to the saved error time
to see how much time has elapsed.
— Jim
> On Feb 15, 2024, at 9:44 AM, James McMahon wrote:
>
> As it turns out lineageStartDate and Queued Duration are very different.
> Without being able to get at Queued Duration as an attribute, it appears we
> cannot RouteOnAttribute to filter thousands in a queue by anything like hours
> they have been in queue.
> Why would this be helpful? Let us say we have an InvokeHttp processor making
> calls to a REST endpoint. We leave for a weekend and return to find 5000
> files in the Failure queue from this processor. It would be most helpful to
> identify the start time and end time of these 5000 failures. We can't do that
> reviewing only the first 100 flowfiles in the queue from the UI.
> One can make an assumption that all of these 5000 flowfiles that failed
> InvokeHttp share a similar range of lineageStartDate, but that will not
> necessarily be true depending on flow complexity.
>
> On Wed, Feb 14, 2024 at 9:49 AM James McMahon > wrote:
>> What a great workaround, thank you once again Mike. I'll put this in and use
>> it now.
>> Jim
>>
>> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser > > wrote:
>>> Hello James,
>>>
>>> I'm not aware of a way to access Queued Duration using expression language,
>>> but you can access the Lineage Duration information. The Getting Started
>>> Guide mentions both entryDate and lineageStartDate as immutable attributes
>>> on all flowfiles. These are numbers of milliseconds since epoch. If you
>>> need them in a readable format, you can use the format() function.
>>>
>>> simple examples:
>>> ${entryDate} = 1707859943778
>>> ${lineageStartDate} = 1707859943778
>>> ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
>>> 21:32:23.778
>>>
>>> -- Mike
>>>
>>>
>>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon >> > wrote:
When we examine the contents of a queue through the UI and select a
flowfile from the resulting list, we see FlowFile Details in the Details
tab. Are those key/values accessible from nifi expression language? I
would like to access Queued Duration. I have a queue that holds flowfiles
with non-successful return codes for calls to REST services, and I want to
route depending on how long these flowfiles have been sitting in my error
queue to isolate the window when the REST service was unavailable.
Thank you for any examples that show how we can access these keys and
values.
Re: Can we access Queued Duration as an attribute?
As it turns out lineageStartDate and Queued Duration are very different.
Without being able to get at Queued Duration as an attribute, it appears we
cannot RouteOnAttribute to filter thousands in a queue by anything like
hours they have been in queue.
Why would this be helpful? Let us say we have an InvokeHttp processor
making calls to a REST endpoint. We leave for a weekend and return to find
5000 files in the Failure queue from this processor. It would be most
helpful to identify the start time and end time of these 5000 failures. We
can't do that reviewing only the first 100 flowfiles in the queue from the
UI.
One can make an assumption that all of these 5000 flowfiles that failed
InvokeHttp share a similar range of lineageStartDate, but that will not
necessarily be true depending on flow complexity.
On Wed, Feb 14, 2024 at 9:49 AM James McMahon wrote:
> What a great workaround, thank you once again Mike. I'll put this in and
> use it now.
> Jim
>
> On Tue, Feb 13, 2024 at 4:41 PM Michael Moser wrote:
>
>> Hello James,
>>
>> I'm not aware of a way to access Queued Duration using expression
>> language, but you can access the Lineage Duration information. The Getting
>> Started Guide mentions both entryDate and lineageStartDate as immutable
>> attributes on all flowfiles. These are numbers of milliseconds since
>> epoch. If you need them in a readable format, you can use the format()
>> function.
>>
>> simple examples:
>> ${entryDate} = 1707859943778
>> ${lineageStartDate} = 1707859943778
>> ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
>> 21:32:23.778
>>
>> -- Mike
>>
>>
>> On Mon, Feb 12, 2024 at 11:38 AM James McMahon
>> wrote:
>>
>>> When we examine the contents of a queue through the UI and select a
>>> flowfile from the resulting list, we see FlowFile Details in the Details
>>> tab. Are those key/values accessible from nifi expression language? I would
>>> like to access Queued Duration. I have a queue that holds flowfiles with
>>> non-successful return codes for calls to REST services, and I want to route
>>> depending on how long these flowfiles have been sitting in my error queue
>>> to isolate the window when the REST service was unavailable.
>>> Thank you for any examples that show how we can access these keys and
>>> values.
>>>
>>
Re: Can we access Queued Duration as an attribute?
What a great workaround, thank you once again Mike. I'll put this in and
use it now.
Jim
On Tue, Feb 13, 2024 at 4:41 PM Michael Moser wrote:
> Hello James,
>
> I'm not aware of a way to access Queued Duration using expression
> language, but you can access the Lineage Duration information. The Getting
> Started Guide mentions both entryDate and lineageStartDate as immutable
> attributes on all flowfiles. These are numbers of milliseconds since
> epoch. If you need them in a readable format, you can use the format()
> function.
>
> simple examples:
> ${entryDate} = 1707859943778
> ${lineageStartDate} = 1707859943778
> ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
> 21:32:23.778
>
> -- Mike
>
>
> On Mon, Feb 12, 2024 at 11:38 AM James McMahon
> wrote:
>
>> When we examine the contents of a queue through the UI and select a
>> flowfile from the resulting list, we see FlowFile Details in the Details
>> tab. Are those key/values accessible from nifi expression language? I would
>> like to access Queued Duration. I have a queue that holds flowfiles with
>> non-successful return codes for calls to REST services, and I want to route
>> depending on how long these flowfiles have been sitting in my error queue
>> to isolate the window when the REST service was unavailable.
>> Thank you for any examples that show how we can access these keys and
>> values.
>>
>
Re: Can we access Queued Duration as an attribute?
Hello James,
I'm not aware of a way to access Queued Duration using expression language,
but you can access the Lineage Duration information. The Getting Started
Guide mentions both entryDate and lineageStartDate as immutable attributes
on all flowfiles. These are numbers of milliseconds since epoch. If you
need them in a readable format, you can use the format() function.
simple examples:
${entryDate} = 1707859943778
${lineageStartDate} = 1707859943778
${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13
21:32:23.778
-- Mike
On Mon, Feb 12, 2024 at 11:38 AM James McMahon wrote:
> When we examine the contents of a queue through the UI and select a
> flowfile from the resulting list, we see FlowFile Details in the Details
> tab. Are those key/values accessible from nifi expression language? I would
> like to access Queued Duration. I have a queue that holds flowfiles with
> non-successful return codes for calls to REST services, and I want to route
> depending on how long these flowfiles have been sitting in my error queue
> to isolate the window when the REST service was unavailable.
> Thank you for any examples that show how we can access these keys and
> values.
>
