RE: [EXT] Re: Metrics on Lineage Duration
Thanks Pierre, though I think you meant to send me: https://pierrevillard.com/2017/05/15/monitoring-nifi-workflow-sla/, which was much more helpful ☺. From: Pierre Villard [mailto:pierre.villard...@gmail.com] Sent: Tuesday, November 07, 2017 8:35 PM To: users@nifi.apache.org Subject: [EXT] Re: Metrics on Lineage Duration Hi Peter, There is not as far as I know. What I usually do is to use ExecuteScript processor to extract the lineage duration of the flow file into an attribute of the flow file where I feel this information is useful for my workflow and I send it to whatever monitoring destination I have. This way I can ensure that mean processing time of a flow file is constant and I can be alerted in case of spurious events. I wrote something about this approach here [1]. [1] https://pierrevillard.com/2017/05/16/monitoring-nifi-ambari-grafana/ Hope this helps, Pierre 2017-11-07 12:55 GMT+01:00 Peter Wicks (pwicks) mailto:pwi...@micron.com>>: Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can monitor my flow using this metric? Thanks, Peter
RE: [EXT] Re: Metrics on Lineage Duration
Mark, While Pierre’s solution will probably be more helpful in the long run, this provides an easy way for me to examine the data by hand right now. I didn’t know that metric was on the Status History window, good to now! Thanks, Peter From: Mark Payne [mailto:marka...@hotmail.com] Sent: Tuesday, November 07, 2017 9:26 PM To: users@nifi.apache.org Subject: [EXT] Re: Metrics on Lineage Duration Peter, In the UI you can right-click on a Processor and go to "View Status History" to see a chart of different metrics over time. One of those metrics is the Average Lineage Duration (averaged over a 5-minute period). Of course, you are asking for a reporting task/processor that does this. There's nothing that provides this info directly, but you could always use InvokeHTTP, or something like that to poll the REST API, using the path /nifi-api/flow/processors//status/history Or, another approach that may be valuable would be to use the SiteToSiteProvenanceReportingTask to send Provenance Events via Site-to-Site back to the NiFi instance. For a given Provenance Event, you can get Event Time and Lineage Start Date, so to calculate the Lineage Duration / latency for the FlowFile, you can just use EventTime - LineageStartDate. This approach would be a little different, as it provides lineage duration of each event. So this could be used, for instance, to detect any particular FlowFiles that exceed some SLA or calculate a 90th percentile type of thing, rather than getting an average. Does this help at all? Thanks -Mark On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can monitor my flow using this metric? Thanks, Peter
Re: Metrics on Lineage Duration
Peter, In the UI you can right-click on a Processor and go to "View Status History" to see a chart of different metrics over time. One of those metrics is the Average Lineage Duration (averaged over a 5-minute period). Of course, you are asking for a reporting task/processor that does this. There's nothing that provides this info directly, but you could always use InvokeHTTP, or something like that to poll the REST API, using the path /nifi-api/flow/processors//status/history Or, another approach that may be valuable would be to use the SiteToSiteProvenanceReportingTask to send Provenance Events via Site-to-Site back to the NiFi instance. For a given Provenance Event, you can get Event Time and Lineage Start Date, so to calculate the Lineage Duration / latency for the FlowFile, you can just use EventTime - LineageStartDate. This approach would be a little different, as it provides lineage duration of each event. So this could be used, for instance, to detect any particular FlowFiles that exceed some SLA or calculate a 90th percentile type of thing, rather than getting an average. Does this help at all? Thanks -Mark On Nov 7, 2017, at 6:55 AM, Peter Wicks (pwicks) mailto:pwi...@micron.com>> wrote: Is there a reporting task/processor that will let me aggregate Lineage Duration’s for a point in time so I can monitor my flow using this metric? Thanks, Peter
Re: Metrics on Lineage Duration
Hi Peter, There is not as far as I know. What I usually do is to use ExecuteScript processor to extract the lineage duration of the flow file into an attribute of the flow file where I feel this information is useful for my workflow and I send it to whatever monitoring destination I have. This way I can ensure that mean processing time of a flow file is constant and I can be alerted in case of spurious events. I wrote something about this approach here [1]. [1] https://pierrevillard.com/2017/05/16/monitoring-nifi-ambari-grafana/ Hope this helps, Pierre 2017-11-07 12:55 GMT+01:00 Peter Wicks (pwicks) : > Is there a reporting task/processor that will let me aggregate Lineage > Duration’s for a point in time so I can monitor my flow using this metric? > > > > Thanks, > > Peter >
Metrics on Lineage Duration
Is there a reporting task/processor that will let me aggregate Lineage Duration's for a point in time so I can monitor my flow using this metric? Thanks, Peter