achristianson opened a new pull request #3681: NIFI-6510 - Analytics framework URL: https://github.com/apache/nifi/pull/3681 #### Description of PR ``` Currently NiFi has lots of metrics available for areas including jvm and flow component usage (via component status) as well as provenance data which NiFi makes available either through the UI or reporting tasks (for consumption by other systems). Past discussions in the community cite users shipping this data to applications such as Prometheus, ELK stacks, or Ambari metrics for further analysis in order to capture/review performance issues, detect anomalies, and send alerts or notifications. These systems are efficient in capturing and helping to analyze these metrics however it requires customization work and knowledge of NiFi operations to provide meaningful analytics within a flow context. In speaking with Matt Burgess and Andy Christianson on this topic we feel that there is an opportunity to introduce an analytics framework that could provide users reasonable predictions on key performance indicators for flows, such as back pressure and flow rate, to help administrators improve operational management of NiFi clusters. This framework could offer several key features: - Provide a flexible internal analytics engine and model api which supports the addition of or enhancement to onboard models - Support integration of remote or cloud based ML models - Support both traditional and online (incremental) learning methods - Provide support for model caching (perhaps later inclusion into a model repository or registry) - UI enhancements to display prediction information either in existing summary data, new data visualizations, or directly within the flow/canvas (where applicable) For an initial target we thought that back pressure prediction would be a good starting point for this initiative, given that back pressure detection is a key indicator of flow performance and many of the metrics currently available would provide enough data points to create a reasonable performing model. We have some ideas on how this could be achieved however we wanted to discuss this more with the community to get thoughts about tackling this work, especially if there are specific use cases or other factors that should be considered This closes NIFI-6510. ``` ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on both JDK 8 and JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
