[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923774#comment-16923774 ]
ASF subversion and git services commented on NIFI-6510: ------------------------------------------------------- Commit 9292005a723c308978894549eb1edf0e3152ac1b in nifi's branch refs/heads/analytics-framework from Rob Fellows [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9292005 ] NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697) > Predictive Analytics for NiFi Metrics > ------------------------------------- > > Key: NIFI-6510 > URL: https://issues.apache.org/jira/browse/NIFI-6510 > Project: Apache NiFi > Issue Type: Improvement > Reporter: Andrew Christianson > Assignee: Yolanda M. Davis > Priority: Major > Fix For: 1.10.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > From Yolanda's email to the list: > > {noformat} > Currently NiFi has lots of metrics available for areas including jvm and flow > component usage (via component status) as well as provenance data which NiFi > makes available either through the UI or reporting tasks (for consumption by > other systems). Past discussions in the community cite users shipping this > data to applications such as Prometheus, ELK stacks, or Ambari metrics for > further analysis in order to capture/review performance issues, detect > anomalies, and send alerts or notifications. These systems are efficient in > capturing and helping to analyze these metrics however it requires > customization work and knowledge of NiFi operations to provide meaningful > analytics within a flow context. > In speaking with Matt Burgess and Andy Christianson on this topic we feel > that there is an opportunity to introduce an analytics framework that could > provide users reasonable predictions on key performance indicators for flows, > such as back pressure and flow rate, to help administrators improve > operational management of NiFi clusters. This framework could offer several > key features: > - Provide a flexible internal analytics engine and model api which supports > the addition of or enhancement to onboard models > - Support integration of remote or cloud based ML models > - Support both traditional and online (incremental) learning methods > - Provide support for model caching (perhaps later inclusion into a model > repository or registry) > - UI enhancements to display prediction information either in existing > summary data, new data visualizations, or directly within the flow/canvas > (where applicable) > For an initial target we thought that back pressure prediction would be a > good starting point for this initiative, given that back pressure detection > is a key indicator of flow performance and many of the metrics currently > available would provide enough data points to create a reasonable performing > model. We have some ideas on how this could be achieved however we wanted to > discuss this more with the community to get thoughts about tackling this > work, especially if there are specific use cases or other factors that should > be considered.{noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)