[jira] [Comment Edited] (NIFI-6498) XMLTransform processor: XSLT doesn't give detail error message with line number
[
https://issues.apache.org/jira/browse/NIFI-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922485#comment-16922485
]
Nico Verwer edited comment on NIFI-6498 at 9/10/19 6:50 AM:
If someone picks this up: Could you please move the version of Saxon-HE to
9.9.1-4?
I removed the earlier suggestion to set a Saxon property, which is better done
through the processor's properties.
was (Author: nverwer):
If someone picks this up: Could you please move the version of Saxon-HE to
9.9.1-4?
And also do
{code:java}
factory.setFeature("http://saxon.sf.net/feature/allow-external-functions";,
true);{code}
> XMLTransform processor: XSLT doesn't give detail error message with line
> number
> ---
>
> Key: NIFI-6498
> URL: https://issues.apache.org/jira/browse/NIFI-6498
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.9.2
>Reporter: Raymond
>Priority: Major
>
> When something is wrong with the XSLT in the XMLTransform processor you get
> an error, but this is only a general error without any details or the line
> number where the parsing error occurs. For example when using Saxon from
> example I get the following error:
> Error on line 17 column 5 SXXP0003: Error reported by XML parser: The element
> type "xsl:tomplate" must be terminated by the matching end-tag
> "".
> When using NiFi the following error is printed:
> 11:07:43 CEST
> ERROR
> 2b4c1dab-3a87-175f-f817-3f3c72af4102
> TransformXml[id=2b4c1dab-3a87-175f-f817-3f3c72af4102] Unable to transform
> StandardFlowFileRecord[uuid=c4dde8e5-89b0-4f29-8f2d-a97a009cf843,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1564391260760-1, container=default,
> section=1], offset=0,
> length=6380],offset=0,name=c4dde8e5-89b0-4f29-8f2d-a97a009cf843,size=6380]
> due to IOException thrown from
> TransformXml[id=2b4c1dab-3a87-175f-f817-3f3c72af4102]: java.io.IOException:
> java.util.concurrent.ExecutionException:
> javax.xml.transform.TransformerConfigurationException:
> net.sf.saxon.s9api.SaxonApiException: Stylesheet compilation failed: 1 error
> reported: org.apache.nifi.processor.exception.ProcessException: IOException
> thrown from TransformXml[id=2b4c1dab-3a87-175f-f817-3f3c72af4102]:
> java.io.IOException: java.util.concurrent.ExecutionException:
> javax.xml.transform.TransformerConfigurationException:
> net.sf.saxon.s9api.SaxonApiException: Stylesheet compilation failed: 1 error
> reported
> The error in NiFi gives almost no information for debugging. Would be a big
> improvement if the Saxon error message is catched and add to the NiFi error
> message (bulletin board).
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (NIFI-6498) XMLTransform processor: XSLT doesn't give detail error message with line number
[
https://issues.apache.org/jira/browse/NIFI-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922473#comment-16922473
]
Nico Verwer edited comment on NIFI-6498 at 9/10/19 6:47 AM:
This response has been updated with refactored code fragments on 2019-09-10.
Saxon uses an ErrorListener when compiling a stylesheet. The
net.sf.saxon.lib.StandardErrorListener writes messages to System.err, if
nothing else is specified. Where they end up is unclear to me.
To send the error messages to Nifi's log-file, change the following in
TransformXML.java:
{code:java}
private Templates newTemplates(final ProcessContext context, final String
path) throws TransformerConfigurationException, LookupFailureException {
final ComponentLog logger = getLogger();
final Boolean secureProcessing =
context.getProperty(SECURE_PROCESSING).asBoolean();
TransformerFactory factory = TransformerFactory.newInstance();
final boolean isFilename = context.getProperty(XSLT_FILE_NAME).isSet();
if (secureProcessing) {
factory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
// don't be overly DTD-unfriendly forcing
http://apache.org/xml/features/disallow-doctype-decl
factory.setFeature("http://saxon.sf.net/feature/parserFeature?uri=http://xml.org/sax/features/external-parameter-entities";,
false);
factory.setFeature("http://saxon.sf.net/feature/parserFeature?uri=http://xml.org/sax/features/external-general-entities";,
false);
}// Set all dynamic properties that are boolean-valued Saxon
features,
// for example "http://saxon.sf.net/feature/allow-external-functions";.
for (final Map.Entry entry :
context.getProperties().entrySet()) {
if (entry.getKey().isDynamic() &&
entry.getKey().getName().startsWith("http://saxon.sf.net/";)) {
factory.setFeature(entry.getKey().getName(),
Boolean.parseBoolean(entry.getValue()));
}
}ErrorListener errorListener = new ErrorListener() {
@Override
public void warning(TransformerException exception) throws
TransformerException {
logger.warn(exception.getMessageAndLocation());
}
@Override
public void error(TransformerException exception) throws
TransformerException {
logger.error(exception.getMessageAndLocation());
}
@Override
public void fatalError(TransformerException exception) throws
TransformerException {
logger.log(LogLevel.FATAL, exception.getMessageAndLocation());
}
};
factory.setErrorListener(errorListener);if(isFilename) {
return factory.newTemplates(new StreamSource(path));
} else {
final String coordinateKey =
lookupService.get().getRequiredKeys().iterator().next();
final Optional attributeValue =
lookupService.get().lookup(Collections.singletonMap(coordinateKey, path));
if (attributeValue.isPresent() &&
StringUtils.isNotBlank(attributeValue.get())) {
return factory.newTemplates(new StreamSource(new
ByteArrayInputStream(attributeValue.get().getBytes(StandardCharsets.UTF_8;
} else {
throw new TransformerConfigurationException("No XSLT definition
is associated to " + path + " in the lookup controller service.");
}
}
}
{code}
The important part is the addition of an ErrorListener. Then, exceptions are
caught and the collected messages are added to the exception.
I have also added the possibility to set binary Saxon properties like
"http://saxon.sf.net/feature/allow-external-functions"; from the processor's
properties.
The above will send compile-time messages to the Nifi logging, but we need to
do something similar for run-time messages emitted by . This
requires the following change inside the onTrigger() method:
{code:java}
public void process(final InputStream rawIn, final OutputStream
out) throws IOException {
try (final InputStream in = new BufferedInputStream(rawIn))
{
final Templates templates;
if (cache != null) {
templates = cache.get(path);
} else {
templates = newTemplates(context, path);
}final Transformer transformer
= templates.newTransformer();
transformer.setOutputProperty(OutputKeys.INDENT,
(indentOutput ? "yes" : "no"));
setMessageLogger(transformer);// Pass all dynamic
properties to the transformer
for (final Map.Entry entry
: context.getProperties().entryS
[GitHub] [nifi] sakina13789 opened a new pull request #3716: Adding infinispan processor
sakina13789 opened a new pull request #3716: Adding infinispan processor URL: https://github.com/apache/nifi/pull/3716 Added Infinispan cache server support. It has 2 processors 1. PutInInfinispan It puts value in cache server against the given cache key 2. GetFromInfinispan It gets the value from cache server against the given cache key This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3…
belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3… URL: https://github.com/apache/nifi/pull/3643#issuecomment-529724982 ``` [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project nifi-hive-processors: Compilation failure [ERROR] error: error reading /home/travis/.m2/repository/org/apache/hive/hive-exec/1.2.1/hive-exec-1.2.1.jar; error in opening zip file ``` Doesn't seem related to the patch. Can someone please kick off build again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] belugabehr commented on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3…
belugabehr commented on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3… URL: https://github.com/apache/nifi/pull/3643#issuecomment-529724982 ```[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project nifi-hive-processors: Compilation failure [ERROR] error: error reading /home/travis/.m2/repository/org/apache/hive/hive-exec/1.2.1/hive-exec-1.2.1.jar; error in opening zip file``` Doesn't seem related to the patch. Can someone please kick off build again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3…
belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3… URL: https://github.com/apache/nifi/pull/3643#issuecomment-529693719 I think that upgrading to Hadoop 3.2 broke an implicit dependency that the NiFi module had on `commons-lang` This dependency came in with Hadoop 3.0. I've added in the dependency to NiFi explicitly. It's clearly using the library. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3…
belugabehr edited a comment on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3… URL: https://github.com/apache/nifi/pull/3643#issuecomment-529693719 I think that upgrading to Hadoop 3.2 broke an implicit dependency that the NiFi module had on `commons-lang`. I've added in the dependency to NiFi explicitly. It's clearly using the library. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] belugabehr commented on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3…
belugabehr commented on issue #3643: NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3… URL: https://github.com/apache/nifi/pull/3643#issuecomment-529693719 I think that upgrading to Hadoop 3.2 removed an implicit dependency that the NiFi module had on `commons-lang`. I've added in the dependency explicitly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] thenatog edited a comment on issue #3715: NIFI-6578 - Upgraded zookeeper framework version. Some code changes r…
thenatog edited a comment on issue #3715: NIFI-6578 - Upgraded zookeeper framework version. Some code changes r… URL: https://github.com/apache/nifi/pull/3715#issuecomment-529679958 Looks like @markap14 @jtstorck are two people who have touched Zookeeper and Zookeeper migrator before. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] thenatog commented on issue #3715: NIFI-6578 - Upgraded zookeeper framework version. Some code changes r…
thenatog commented on issue #3715: NIFI-6578 - Upgraded zookeeper framework version. Some code changes r… URL: https://github.com/apache/nifi/pull/3715#issuecomment-529679958 @markap14 @jtstorck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] thenatog opened a new pull request #3715: NIFI-6578 - Upgraded zookeeper framework version. Some code changes r…
thenatog opened a new pull request #3715: NIFI-6578 - Upgraded zookeeper
framework version. Some code changes r…
URL: https://github.com/apache/nifi/pull/3715
…equired. This change also required a change to the embedded zookeeper.
NIFI-6578 - Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit.
Updated tests to allow for new znode /zookeeper/config
NIFI-6578 - Added documentation changes to reflect changes to the
zookeeper.properties file going foward.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] scottyaslan commented on issue #3710: NIFI-6641 - support read only mode for parameter context dialog
scottyaslan commented on issue #3710: NIFI-6641 - support read only mode for parameter context dialog URL: https://github.com/apache/nifi/pull/3710#issuecomment-529650370 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mcgilman commented on issue #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field
mcgilman commented on issue #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field URL: https://github.com/apache/nifi/pull/3706#issuecomment-529646516 Will review... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6644) UI: Account for user permissions when showing the option to convert props to params
[ https://issues.apache.org/jira/browse/NIFI-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6644: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > UI: Account for user permissions when showing the option to convert props to > params > --- > > Key: NIFI-6644 > URL: https://issues.apache.org/jira/browse/NIFI-6644 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Robert Fellows >Assignee: Robert Fellows >Priority: Major > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > A user that does not have write permissions to the appropriate parameter > context should not be able to convert a property to a parameter. The option > should not be available in the property table. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6644) UI: Account for user permissions when showing the option to convert props to params
[ https://issues.apache.org/jira/browse/NIFI-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926072#comment-16926072 ] ASF subversion and git services commented on NIFI-6644: --- Commit 5ddc01eb9f8546a09fabf08890873c4f0e1bdd02 in nifi's branch refs/heads/master from Rob Fellows [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5ddc01e ] NIFI-6644 - Consider param context permissions when deciding to allow property conversion to parameter. NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings. This closes #3713 > UI: Account for user permissions when showing the option to convert props to > params > --- > > Key: NIFI-6644 > URL: https://issues.apache.org/jira/browse/NIFI-6644 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Robert Fellows >Assignee: Robert Fellows >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > A user that does not have write permissions to the appropriate parameter > context should not be able to convert a property to a parameter. The option > should not be available in the property table. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6644) UI: Account for user permissions when showing the option to convert props to params
[ https://issues.apache.org/jira/browse/NIFI-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926073#comment-16926073 ] ASF subversion and git services commented on NIFI-6644: --- Commit 5ddc01eb9f8546a09fabf08890873c4f0e1bdd02 in nifi's branch refs/heads/master from Rob Fellows [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5ddc01e ] NIFI-6644 - Consider param context permissions when deciding to allow property conversion to parameter. NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings. This closes #3713 > UI: Account for user permissions when showing the option to convert props to > params > --- > > Key: NIFI-6644 > URL: https://issues.apache.org/jira/browse/NIFI-6644 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Robert Fellows >Assignee: Robert Fellows >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > A user that does not have write permissions to the appropriate parameter > context should not be able to convert a property to a parameter. The option > should not be available in the property table. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3713: NIFI-6644 - Consider param context permissions when deciding to allow…
mcgilman commented on issue #3713: NIFI-6644 - Consider param context permissions when deciding to allow… URL: https://github.com/apache/nifi/pull/3713#issuecomment-529645620 Thanks @rfellows! This has been merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] asfgit closed pull request #3713: NIFI-6644 - Consider param context permissions when deciding to allow…
asfgit closed pull request #3713: NIFI-6644 - Consider param context permissions when deciding to allow… URL: https://github.com/apache/nifi/pull/3713 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-fds] scottyaslan opened a new pull request #56: [NIFI-6646] untheme confirm dialog close button
scottyaslan opened a new pull request #56: [NIFI-6646] untheme confirm dialog close button URL: https://github.com/apache/nifi-fds/pull/56 Thank you for submitting a contribution to Apache NiFi Flow Design System. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with either NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you ensured that a full build and that the full suite of unit tests is executed via npm run clean:install at the root nifi-fds folder? - [ ] Have you written or updated the Apache NiFi Flow Design System demo application to demonstrate any new functionality, provide examples of usage, and to verify your changes via npm start at the nifi-fds/target folder? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-fds? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-fds? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-6646) fds confirm dialog close button has incorrect color
Scott Aslan created NIFI-6646: - Summary: fds confirm dialog close button has incorrect color Key: NIFI-6646 URL: https://issues.apache.org/jira/browse/NIFI-6646 Project: Apache NiFi Issue Type: Improvement Affects Versions: fds-0.3 Reporter: Scott Aslan Assignee: Scott Aslan Fix For: fds-0.3 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6645) Fix unit tests for StandardParameterContext
[ https://issues.apache.org/jira/browse/NIFI-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-6645: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Fix unit tests for StandardParameterContext > --- > > Key: NIFI-6645 > URL: https://issues.apache.org/jira/browse/NIFI-6645 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6645) Fix unit tests for StandardParameterContext
[ https://issues.apache.org/jira/browse/NIFI-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926027#comment-16926027 ] ASF subversion and git services commented on NIFI-6645: --- Commit f89ea8cf2130348e752286d0e76173f52c3d6912 in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f89ea8c ] NIFI-6645: Fixed problem in unit test for StandardParameterContext because behavior changed; fixed issue in StandardParameterContext around adding a parameter with no description This closes #3714. Signed-off-by: Bryan Bende > Fix unit tests for StandardParameterContext > --- > > Key: NIFI-6645 > URL: https://issues.apache.org/jira/browse/NIFI-6645 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3714: NIFI-6645: Fixed problem in unit test for StandardParameterContext be…
asfgit closed pull request #3714: NIFI-6645: Fixed problem in unit test for StandardParameterContext be… URL: https://github.com/apache/nifi/pull/3714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] bbende commented on issue #3714: NIFI-6645: Fixed problem in unit test for StandardParameterContext be…
bbende commented on issue #3714: NIFI-6645: Fixed problem in unit test for StandardParameterContext be… URL: https://github.com/apache/nifi/pull/3714#issuecomment-529627174 Looks good, tests passing for me now, will merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6645) Fix unit tests for StandardParameterContext
[ https://issues.apache.org/jira/browse/NIFI-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-6645: - Fix Version/s: 1.10.0 Assignee: Mark Payne Status: Patch Available (was: Open) > Fix unit tests for StandardParameterContext > --- > > Key: NIFI-6645 > URL: https://issues.apache.org/jira/browse/NIFI-6645 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] markap14 opened a new pull request #3714: NIFI-6645: Fixed problem in unit test for StandardParameterContext be…
markap14 opened a new pull request #3714: NIFI-6645: Fixed problem in unit test
for StandardParameterContext be…
URL: https://github.com/apache/nifi/pull/3714
…cause behavior changed; fixed issue in StandardParameterContext around
adding a parameter with no description
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (NIFI-6645) Fix unit tests for StandardParameterContext
Mark Payne created NIFI-6645: Summary: Fix unit tests for StandardParameterContext Key: NIFI-6645 URL: https://issues.apache.org/jira/browse/NIFI-6645 Project: Apache NiFi Issue Type: Bug Reporter: Mark Payne -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] SandishKumarHN commented on issue #3611: NIFI-6009 ScanKudu Processor
SandishKumarHN commented on issue #3611: NIFI-6009 ScanKudu Processor URL: https://github.com/apache/nifi/pull/3611#issuecomment-529615749 @pvillard31 made changes based on your suggestions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Issue Type: New Feature (was: Improvement)
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
This feature was merged 9/9/2019
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (NIFI-6574) Add check for valid predictionIntervalMillis value
[ https://issues.apache.org/jira/browse/NIFI-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6574. Fix Version/s: 1.10.0 Resolution: Fixed This appears to have been resolved in the context of NIFI-6510 parent PR commits. Setting to resolved > Add check for valid predictionIntervalMillis value > -- > > Key: NIFI-6574 > URL: https://issues.apache.org/jira/browse/NIFI-6574 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Priority: Minor > Fix For: 1.10.0 > > > For backpressure prediction the predictionIntervalMillis should have a > validity check for value provided. If invalid default value should be set > with a warning in logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mattyb149 commented on issue #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
mattyb149 commented on issue #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types URL: https://github.com/apache/nifi/pull/3684#issuecomment-529601617 @tpalfy Are you available to review the latest changes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6511) Specify remote protocol for predictive analytics
[ https://issues.apache.org/jira/browse/NIFI-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6511: --- Parent: (was: NIFI-6510) Issue Type: New Feature (was: Sub-task) > Specify remote protocol for predictive analytics > > > Key: NIFI-6511 > URL: https://issues.apache.org/jira/browse/NIFI-6511 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andrew Christianson >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In order to allow more resource-demanding analytics to be run outside of the > main NiFi process, we need to define a protocol which allows NiFi to send > analytic input data (e.g. NiFi metrics) and receive > predictions/answers/useful values back. > Because the amount of input data to models may be large, the protocol needs > to be lightweight and efficient. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] markap14 commented on issue #3667: NIFI-6582 Add diff-flow-versions command to CLI
markap14 commented on issue #3667: NIFI-6582 Add diff-flow-versions command to CLI URL: https://github.com/apache/nifi/pull/3667#issuecomment-529594088 Thanks @bbende! Was trying this out. Things seem to work well. There is a bit of an inconsistency, here, though. In order to perform a diff of two flows, I wanted to determine the version numbers of a Flow that I have in registry. So I used the `list-flows` command, got the flow identifier, then used the `list-flow-versions` command to get the version numbers. The `list-flow-versions` command needs the `flowIdentifier` provided but not a `bucketIdentifier`. But then when I attempt to perform the diff, I have to provide a bucket identifier. I think we should try to be consistent here and not require the bucket ID, if possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] markap14 commented on issue #3702: NIFI-6636: Fixed ListGCSBucket file duplication error
markap14 commented on issue #3702: NIFI-6636: Fixed ListGCSBucket file duplication error URL: https://github.com/apache/nifi/pull/3702#issuecomment-529585851 Thanks for the update @turcsanyip . All looks good at this point so +1 merged to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6636) ListGCSBucket duplicates files if they arrive not in alphabetical order
[ https://issues.apache.org/jira/browse/NIFI-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925943#comment-16925943 ] ASF subversion and git services commented on NIFI-6636: --- Commit 21a27c8bb00c3f41836c938c3bb59ff428a27ca4 in nifi's branch refs/heads/master from Peter Turcsanyi [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=21a27c8 ] NIFI-6636: Fixed ListGCSBucket file duplication error ListGCSBucket duplicated files if they arrived not in alphabetical order. The set storing the name of the latest blob (which was loaded with the highest timestamp during the previous run of the processor) was cleared too early. Also changed the state persisting logic: it is now saved only once at the end of onTrigger() (similar to ListS3). Some inconsistent state (only blob names without the timestamp) was also saved earlier. This closes #3702. Signed-off-by: Mark Payne > ListGCSBucket duplicates files if they arrive not in alphabetical order > --- > > Key: NIFI-6636 > URL: https://issues.apache.org/jira/browse/NIFI-6636 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > If the files are being uploaded to the bucket NOT in alphabetical order, then > the last loaded file will be reloaded again at the next run of the processor. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6636) ListGCSBucket duplicates files if they arrive not in alphabetical order
[ https://issues.apache.org/jira/browse/NIFI-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-6636: - Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > ListGCSBucket duplicates files if they arrive not in alphabetical order > --- > > Key: NIFI-6636 > URL: https://issues.apache.org/jira/browse/NIFI-6636 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Fix For: 1.10.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > If the files are being uploaded to the bucket NOT in alphabetical order, then > the last loaded file will be reloaded again at the next run of the processor. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3702: NIFI-6636: Fixed ListGCSBucket file duplication error
asfgit closed pull request #3702: NIFI-6636: Fixed ListGCSBucket file duplication error URL: https://github.com/apache/nifi/pull/3702 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] markap14 commented on a change in pull request #3702: NIFI-6636: Fixed ListGCSBucket file duplication error
markap14 commented on a change in pull request #3702: NIFI-6636: Fixed
ListGCSBucket file duplication error
URL: https://github.com/apache/nifi/pull/3702#discussion_r322364290
##
File path:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java
##
@@ -381,40 +384,36 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
// Update state
if (lastModified > maxTimestamp) {
maxTimestamp = lastModified;
-currentKeys.clear();
+maxKeys.clear();
}
if (lastModified == maxTimestamp) {
-currentKeys.add(blob.getName());
+maxKeys.add(blob.getName());
}
-listCount++;
+loadCount++;
}
-blobPages = blobPages.getNextPage();
-commit(context, session, listCount);
-listCount = 0;
-} while (blobPages != null);
-
-currentTimestamp = maxTimestamp;
+commit(context, session, loadCount);
-final long listMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
-getLogger().info("Successfully listed GCS bucket {} in {} millis", new
Object[]{bucket, listMillis});
+blobPage = blobPage.getNextPage();
+} while (blobPage != null);
-if (!commit(context, session, listCount)) {
-if (currentTimestamp > 0) {
-persistState(context);
-}
-getLogger().debug("No new objects in GCS bucket {} to list.
Yielding.", new Object[]{bucket});
+if (maxTimestamp != 0) {
+currentTimestamp = maxTimestamp;
+currentKeys = maxKeys;
+persistState(context);
+} else {
+getLogger().debug("No new objects in GCS bucket {} to load.
Yielding.", new Object[]{bucket});
context.yield();
}
+
+final long listMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
+getLogger().info("Successfully listed GCS bucket {} in {} millis", new
Object[]{bucket, listMillis});
}
-private boolean commit(final ProcessContext context, final ProcessSession
session, int listCount) {
-boolean willCommit = listCount > 0;
-if (willCommit) {
-getLogger().info("Successfully listed {} new files from GCS;
routing to success", new Object[] {listCount});
+private void commit(final ProcessContext context, final ProcessSession
session, int loadCount) {
+if (loadCount > 0) {
+getLogger().info("Successfully loaded {} new files from GCS;
routing to success", new Object[] {loadCount});
Review comment:
Thanks, looks good to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] markap14 commented on a change in pull request #3702: NIFI-6636: Fixed ListGCSBucket file duplication error
markap14 commented on a change in pull request #3702: NIFI-6636: Fixed
ListGCSBucket file duplication error
URL: https://github.com/apache/nifi/pull/3702#discussion_r322364207
##
File path:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java
##
@@ -381,40 +384,36 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
// Update state
if (lastModified > maxTimestamp) {
maxTimestamp = lastModified;
-currentKeys.clear();
+maxKeys.clear();
}
if (lastModified == maxTimestamp) {
-currentKeys.add(blob.getName());
+maxKeys.add(blob.getName());
}
-listCount++;
+loadCount++;
}
-blobPages = blobPages.getNextPage();
-commit(context, session, listCount);
-listCount = 0;
-} while (blobPages != null);
-
-currentTimestamp = maxTimestamp;
+commit(context, session, loadCount);
-final long listMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
-getLogger().info("Successfully listed GCS bucket {} in {} millis", new
Object[]{bucket, listMillis});
+blobPage = blobPage.getNextPage();
+} while (blobPage != null);
-if (!commit(context, session, listCount)) {
-if (currentTimestamp > 0) {
-persistState(context);
-}
-getLogger().debug("No new objects in GCS bucket {} to list.
Yielding.", new Object[]{bucket});
+if (maxTimestamp != 0) {
+currentTimestamp = maxTimestamp;
+currentKeys = maxKeys;
+persistState(context);
+} else {
+getLogger().debug("No new objects in GCS bucket {} to load.
Yielding.", new Object[]{bucket});
context.yield();
}
+
+final long listMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
+getLogger().info("Successfully listed GCS bucket {} in {} millis", new
Object[]{bucket, listMillis});
}
-private boolean commit(final ProcessContext context, final ProcessSession
session, int listCount) {
-boolean willCommit = listCount > 0;
-if (willCommit) {
-getLogger().info("Successfully listed {} new files from GCS;
routing to success", new Object[] {listCount});
+private void commit(final ProcessContext context, final ProcessSession
session, int loadCount) {
+if (loadCount > 0) {
+getLogger().info("Successfully loaded {} new files from GCS;
routing to success", new Object[] {loadCount});
session.commit();
-persistState(context);
Review comment:
Ah, OK, I didn't realize that was the case. I was able to verify this by
performing a listing on a bucket with nearly 100,000 items. After about 40-50
thousand had been listed, i restarted NiFi and upon restart, sure enough it
re-listed everything. I do think that we need to find a way to handle this
better. But for now, this PR does fix a bug and doesn't make anything worse, so
it makes sense to go ahead and merge as-is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (NIFI-6578) Upgrade zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/NIFI-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925924#comment-16925924 ] Nathan Gough edited comment on NIFI-6578 at 9/9/19 5:03 PM: On implementing this upgrade, I've found that there has been a small change to the way the zookeeper.properties file is handled, as per NIFI-6624. I have updated the embedded zookeeper.properties file that gets deployed, but users will likely need to change their existing properties files when using embedded zookeeper. We will also need to update documentation. was (Author: thenatog): On implementing this upgrade, I've found that there has been a small change to the way the zookeeper.properties file is handled, as per [NIFI-6624 . I have updated the embedded zookeeper.properties file that gets deployed, but users will likely need to change their existing properties files when using embedded zookeeper. We will also need to update documentation. > Upgrade zookeeper to 3.5.5 > -- > > Key: NIFI-6578 > URL: https://issues.apache.org/jira/browse/NIFI-6578 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.9.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 1.10.0 > > > Upgrade zookeeper dependencies to 3.5.5 and test. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6578) Upgrade zookeeper to 3.5.5
[ https://issues.apache.org/jira/browse/NIFI-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925924#comment-16925924 ] Nathan Gough commented on NIFI-6578: On implementing this upgrade, I've found that there has been a small change to the way the zookeeper.properties file is handled, as per [NIFI-6624 . I have updated the embedded zookeeper.properties file that gets deployed, but users will likely need to change their existing properties files when using embedded zookeeper. We will also need to update documentation. > Upgrade zookeeper to 3.5.5 > -- > > Key: NIFI-6578 > URL: https://issues.apache.org/jira/browse/NIFI-6578 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.9.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 1.10.0 > > > Upgrade zookeeper dependencies to 3.5.5 and test. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6624) Automatically migrate old embedded zookeeper.properties files to the new format
[ https://issues.apache.org/jira/browse/NIFI-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Gough updated NIFI-6624: --- Parent: NIFI-6578 Issue Type: Sub-task (was: Improvement) > Automatically migrate old embedded zookeeper.properties files to the new > format > --- > > Key: NIFI-6624 > URL: https://issues.apache.org/jira/browse/NIFI-6624 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Affects Versions: 1.9.2 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 1.10.0 > > > The 3.5.x Zookeeper properties file has changed: > [https://zookeeper.apache.org/doc/r3.5.2-alpha/zookeeperReconfig.html#sc_reconfig_clientport] > As a result, the current embedded zookeeper bundled with NiFi will require a > minor change to the zookeeper.properties file. We could potentially do this > automatically by removing the clientPort line and appending the value to the > end of the server string/s. > The assumption here would be that the client port is currently configured the > same across all servers. If not, we would not be able to auto-migrate the > change, and we would need to fail the start and get the user to configure the > zookeeper.properties file correctly across all nodes. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3713: NIFI-6644 - Consider param context permissions when deciding to allow…
mcgilman commented on issue #3713: NIFI-6644 - Consider param context permissions when deciding to allow… URL: https://github.com/apache/nifi/pull/3713#issuecomment-529543275 Will review... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6568) Surface time-to-back-pressure and initial predictions in the UI.
[ https://issues.apache.org/jira/browse/NIFI-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925818#comment-16925818 ] ASF subversion and git services commented on NIFI-6568: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925826#comment-16925826 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925829#comment-16925829 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925830#comment-16925830 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925834#comment-16925834 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925836#comment-16925836 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925828#comment-16925828 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925822#comment-16925822 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925824#comment-16925824 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925831#comment-16925831 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925827#comment-16925827 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925832#comment-16925832 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925823#comment-16925823 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925835#comment-16925835 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925837#comment-16925837 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925838#comment-16925838 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925825#comment-16925825 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6568) Surface time-to-back-pressure and initial predictions in the UI.
[ https://issues.apache.org/jira/browse/NIFI-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925817#comment-16925817 ] ASF subversion and git services commented on NIFI-6568: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6586) Comments and Documentation for Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925816#comment-16925816 ] ASF subversion and git services commented on NIFI-6586: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925810#comment-16925810 ] ASF subversion and git services commented on NIFI-6566: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925819#comment-16925819 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925806#comment-16925806 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6150) Content Repo not getting cleaned up
[ https://issues.apache.org/jira/browse/NIFI-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925802#comment-16925802 ] ASF subversion and git services commented on NIFI-6150: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925809#comment-16925809 ] ASF subversion and git services commented on NIFI-6566: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925820#comment-16925820 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6150) Content Repo not getting cleaned up
[ https://issues.apache.org/jira/browse/NIFI-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925804#comment-16925804 ] ASF subversion and git services commented on NIFI-6150: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6585) Update tests to decouple model validation from status analytics & engine validation
[ https://issues.apache.org/jira/browse/NIFI-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925814#comment-16925814 ] ASF subversion and git services commented on NIFI-6585: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925805#comment-16925805 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6586) Comments and Documentation for Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925815#comment-16925815 ] ASF subversion and git services commented on NIFI-6586: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6150) Content Repo not getting cleaned up
[ https://issues.apache.org/jira/browse/NIFI-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925803#comment-16925803 ] ASF subversion and git services commented on NIFI-6150: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925813#comment-16925813 ] ASF subversion and git services commented on NIFI-6566: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925808#comment-16925808 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925821#comment-16925821 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925812#comment-16925812 ] ASF subversion and git services commented on NIFI-6566: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925791#comment-16925791 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925786#comment-16925786 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925789#comment-16925789 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925790#comment-16925790 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925792#comment-16925792 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925797#comment-16925797 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925787#comment-16925787 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925799#comment-16925799 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925795#comment-16925795 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925801#comment-16925801 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925793#comment-16925793 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925794#comment-16925794 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925796#comment-16925796 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925788#comment-16925788 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925798#comment-16925798 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925800#comment-16925800 ] ASF subversion and git services commented on NIFI-6510: --- Commit 8a8b9c1d086ac41b10647f09bdd7d8174a921de5 in nifi's branch refs/heads/master from Andy I. Christianson [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8a8b9c1 ] NIFI-6510 - Analytics framework (#3681) * NIFI-6510 Implement initial analytic engine * NIFI-6510 Implemented basic linear regression model for queue counts * NIFI-6510 Initial analytics REST endpoint and supporting objects * NIFI-6510 Connect the dots for StatusAnalytics -> API * NIFI-6510 Added poc engine with prediction model caching (cherry picked from commit e013b91) DFA-9 - updated logging and corrected logic for checking if not in backpressure (cherry picked from commit a1f8e70) * NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics (cherry picked from commit 050e0fc) (cherry picked from commit 9fd365f) * NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id (cherry picked from commit 14854ff) DFA-9 - reduced snapshot interval to 1 minute (cherry picked from commit 36abb0a) * NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions * NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly * NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly" This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1. * NIFI-6510 Added prediction fields for use by UI, still need to be populated * NIFI-6510 Analytics Framework Introduction (#10) * DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model. DFA-9 - added logging * DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval * DFA-9 - checkstyle fixes * NIFI-6510 Add prediction percent values and predicted interval seconds (cherry picked from commit e60015d) * NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not. (cherry picked from commit 6fae058) * NIFI-6510 Added tests for engine (cherry picked from commit 6d7a13b) * NIFI-6150 Added tests for connection status analytics class, corrected variable names (cherry picked from commit 58c7c81) * NIFI-6150 Make checkstyle happy (cherry picked from commit b6e35ac) * NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing * NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics * NIFI-6510 Adjusted interval and incorporated R-squared check Updates to support multiple variables for features, clearing cached regression model based on r-squared values Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support). Ratcheck fixes Added test for SimpleRegression. Minor fix for OLS model fixed test errors fixed checkstyle errors (cherry picked from commit fab411b) * NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11) * NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties NIFI-6566 - changes to allow scoring configurations for model in nifi.properties NIFI-6566 - added default implementation value to NiFiProperties NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics Signed-off-by: Matthew Burgess This closes #3663 * NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type * NIFI-6586 - documentation and comments This closes NIFI-6586 Signed-off-by: Andrew I. Christianson * NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI * Add multi-line tooltips with detail for connection queue back pressure graphics. * Add estimated time to back pressure to connections summary table. * Add back pressure prediction ticks. * add moment.js to format predicted time to back pressure * tweak summary table headings to match data displayed. re-order connection summary columns * NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment. * NIFI-6510 - add an enable/disable property for analytics * NIFI-6510 - documentation updates for enable
[GitHub] [nifi] achristianson merged pull request #3681: NIFI-6510 - Analytics framework
achristianson merged pull request #3681: NIFI-6510 - Analytics framework URL: https://github.com/apache/nifi/pull/3681 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] markap14 commented on issue #3681: NIFI-6510 - Analytics framework
markap14 commented on issue #3681: NIFI-6510 - Analytics framework URL: https://github.com/apache/nifi/pull/3681#issuecomment-529536876 Thanks guys. All of my major concerns are addressed at this point, I think. I'm a +1 to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925766#comment-16925766
]
ASF subversion and git services commented on NIFI-6510:
---
Commit 16f7c37f05aa722a748d309297c4b0b92675c153 in nifi's branch
refs/heads/analytics-framework from Andrew I. Christianson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=16f7c37 ]
NIFI-6510 Rip out useless members
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [nifi] markap14 commented on issue #3698: WIP: NIFI-6628: Separate out logging of extensions vs. nifi framework
markap14 commented on issue #3698: WIP: NIFI-6628: Separate out logging of extensions vs. nifi framework URL: https://github.com/apache/nifi/pull/3698#issuecomment-529521247 Thanks @adarmiento . I think this is getting closer. It looks like the latest commit puts `org.apache.nifi` into the framework, log, though, which I think is still not what we want, because we have some processors, services, etc. that even are in that package. For example, `GetHTMLElement`, `ModifyHTMLElement`, and `PutHTMLElement` are all in the `org.apache.nifi` package. I think we have to be even more specific. I would look at the `nifi-nar-bundles/nifi-framework-bundle/nifi-framework` module and its sub-modules. Especially `nifi-framework-core`, `nifi-framework-core-api`, `nifi-framework-clustering` and pick out any specific packages in those. For example, the `org.apache.nifi.controller` package contains a lot of the framework specific classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] adarmiento edited a comment on issue #3700: NIFI-6638: Empty multiple queues at once at different flow levels
adarmiento edited a comment on issue #3700: NIFI-6638: Empty multiple queues at once at different flow levels URL: https://github.com/apache/nifi/pull/3700#issuecomment-529461793 Are we sure we need that many functionalities? > selected queue: empty the selected queue > selected queues: empty the selected queues > current process group: empty all queues inside the current process group > current process group (recursive): empty all queues inside the current process group recursively, > which means all queues inside the current process group, and its sub process groups, and their sub process groups [and so on] will be emptied > selected process group: empty all queues inside the selected process group > selected process group (recursive): empty all queues inside the selected process group recursively > selected process groups: empty all queues inside the selected process groups > selected process groups (recursive): empty all queues inside the selected process groups recursively Are there use cases in which you are interested into emptying all queue of a processor group which is not the current one AND at the same time we do not want to delete the queue recursively? I think that the most useful features are: > selected queue: empty the selected queue > selected queues: empty the selected queues > current process group (recursive): empty all queues inside the current process group recursively And therefore I'd start by introducing these ones (which are by chance the most comprehensive, so it would be easy to add the other ones in a second moment) and potentially add the other cases with another PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] adarmiento commented on issue #3700: NIFI-6638: Empty multiple queues at once at different flow levels
adarmiento commented on issue #3700: NIFI-6638: Empty multiple queues at once at different flow levels URL: https://github.com/apache/nifi/pull/3700#issuecomment-529520440 Some minor style fixes in the suggestions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
