RE: [EXT] Re: Metrics on Lineage Duration

[Peter Wicks (pwicks)]

Thanks Pierre, though I think you meant to send me: 
https://pierrevillard.com/2017/05/15/monitoring-nifi-workflow-sla/, which was 
much more helpful ☺.

From: Pierre Villard [mailto:pierre.villard...@gmail.com]
Sent: Tuesday, November 07, 2017 8:35 PM
To: users@nifi.apache.org
Subject: [EXT] Re: Metrics on Lineage Duration

Hi Peter,
There is not as far as I know. What I usually do is to use ExecuteScript 
processor to extract the lineage duration of the flow file into an attribute of 
the flow file where I feel this information is useful for my workflow and I 
send it to whatever monitoring destination I have. This way I can ensure that 
mean processing time of a flow file is constant and I can be alerted in case of 
spurious events. I wrote something about this approach here [1].

[1] https://pierrevillard.com/2017/05/16/monitoring-nifi-ambari-grafana/

Hope this helps,
Pierre


2017-11-07 12:55 GMT+01:00 Peter Wicks (pwicks) 
<pwi...@micron.com<mailto:pwi...@micron.com>>:
Is there a reporting task/processor that will let me aggregate Lineage 
Duration’s for a point in time so I can monitor my flow using this metric?

Thanks,
  Peter

RE: [EXT] Re: Metrics on Lineage Duration

[Peter Wicks (pwicks)]

Mark,

While Pierre’s solution will probably be more helpful in the long run, this 
provides an easy way for me to examine the data by hand right now. I didn’t 
know that metric was on the Status History window, good to now!

Thanks,
  Peter

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Tuesday, November 07, 2017 9:26 PM
To: users@nifi.apache.org
Subject: [EXT] Re: Metrics on Lineage Duration

Peter,

In the UI you can right-click on a Processor and go to "View Status History" to 
see a chart of different metrics
over time. One of those metrics is the Average Lineage Duration (averaged over 
a 5-minute period).
Of course, you are asking for a reporting task/processor that does this. 
There's nothing that provides this info
directly, but you could always use InvokeHTTP, or something like that to poll 
the REST API, using the path
/nifi-api/flow/processors//status/history

Or, another approach that may be valuable would be to use the 
SiteToSiteProvenanceReportingTask to send
Provenance Events via Site-to-Site back to the NiFi instance. For a given 
Provenance Event, you can get Event Time
and Lineage Start Date, so to calculate the Lineage Duration / latency for the 
FlowFile, you can just use
EventTime - LineageStartDate. This approach would be a little different, as it 
provides lineage duration of each event.
So this could be used, for instance, to detect any particular FlowFiles that 
exceed some SLA or calculate a 90th percentile
type of thing, rather than getting an average.

Does this help at all?

Thanks
-Mark


On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) 
<pwi...@micron.com<mailto:pwi...@micron.com>> wrote:

Is there a reporting task/processor that will let me aggregate Lineage 
Duration’s for a point in time so I can monitor my flow using this metric?

Thanks,
  Peter

Re: Metrics on Lineage Duration

[Mark Payne]

Peter,

In the UI you can right-click on a Processor and go to "View Status History" to 
see a chart of different metrics
over time. One of those metrics is the Average Lineage Duration (averaged over 
a 5-minute period).
Of course, you are asking for a reporting task/processor that does this. 
There's nothing that provides this info
directly, but you could always use InvokeHTTP, or something like that to poll 
the REST API, using the path
/nifi-api/flow/processors//status/history

Or, another approach that may be valuable would be to use the 
SiteToSiteProvenanceReportingTask to send
Provenance Events via Site-to-Site back to the NiFi instance. For a given 
Provenance Event, you can get Event Time
and Lineage Start Date, so to calculate the Lineage Duration / latency for the 
FlowFile, you can just use
EventTime - LineageStartDate. This approach would be a little different, as it 
provides lineage duration of each event.
So this could be used, for instance, to detect any particular FlowFiles that 
exceed some SLA or calculate a 90th percentile
type of thing, rather than getting an average.

Does this help at all?

Thanks
-Mark


On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) 
> wrote:

Is there a reporting task/processor that will let me aggregate Lineage 
Duration’s for a point in time so I can monitor my flow using this metric?

Thanks,
  Peter

Re: Metrics on Lineage Duration

[Pierre Villard]

Hi Peter,

There is not as far as I know. What I usually do is to use ExecuteScript
processor to extract the lineage duration of the flow file into an
attribute of the flow file where I feel this information is useful for my
workflow and I send it to whatever monitoring destination I have. This way
I can ensure that mean processing time of a flow file is constant and I can
be alerted in case of spurious events. I wrote something about this
approach here [1].

[1] https://pierrevillard.com/2017/05/16/monitoring-nifi-ambari-grafana/

Hope this helps,
Pierre


2017-11-07 12:55 GMT+01:00 Peter Wicks (pwicks) :

> Is there a reporting task/processor that will let me aggregate Lineage
> Duration’s for a point in time so I can monitor my flow using this metric?
>
>
>
> Thanks,
>
>   Peter
>

Metrics on Lineage Duration

[Peter Wicks (pwicks)]

Is there a reporting task/processor that will let me aggregate Lineage 
Duration's for a point in time so I can monitor my flow using this metric?

Thanks,
  Peter