[jira] [Created] (NIFI-11957) Add accepted query parameter values to documentation for Flow getMetrics API
Yolanda M. Davis created NIFI-11957: --- Summary: Add accepted query parameter values to documentation for Flow getMetrics API Key: NIFI-11957 URL: https://issues.apache.org/jira/browse/NIFI-11957 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.23.0 Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Documentation exists for the getMetrics api in the Flow Resource, however for queryParameters there is no information provided for accepted values (specifically for the includedRegistries). This should be updated so that users know the appropriate values for the endpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11899) Existing nifi_bulletin metric does not reflect updated state once bulletin is resolved
[ https://issues.apache.org/jira/browse/NIFI-11899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-11899: Fix Version/s: 1.latest 2.latest Status: Patch Available (was: In Progress) > Existing nifi_bulletin metric does not reflect updated state once bulletin is > resolved > -- > > Key: NIFI-11899 > URL: https://issues.apache.org/jira/browse/NIFI-11899 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.23.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Minor > Labels: metrics > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > When bulletins are generated, the existing nifi_bulletin metric will > demonstrate the existence of bulletin by registering the metric with a value > of 1 and labels reflecting the specific information for the bulletin. However > once a bulletin no longer exists or is resolved the value persists in the > metrics registry, giving a false sense of the bulletin being active. The > goal of this fix is to properly update the registry to reflect currently > active/inactive bulletins. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11899) Existing nifi_bulletin metric does not reflect updated state once bulletin is resolved
Yolanda M. Davis created NIFI-11899: --- Summary: Existing nifi_bulletin metric does not reflect updated state once bulletin is resolved Key: NIFI-11899 URL: https://issues.apache.org/jira/browse/NIFI-11899 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.23.0 Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis When bulletins are generated, the existing nifi_bulletin metric will demonstrate the existence of bulletin by registering the metric with a value of 1 and labels reflecting the specific information for the bulletin. However once a bulletin no longer exists or is resolved the value persists in the metrics registry, giving a false sense of the bulletin being active. The goal of this fix is to properly update the registry to reflect currently active/inactive bulletins. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11622) Create Bulletin Counter Metric
Yolanda M. Davis created NIFI-11622: --- Summary: Create Bulletin Counter Metric Key: NIFI-11622 URL: https://issues.apache.org/jira/browse/NIFI-11622 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.21.0 Reporter: Yolanda M. Davis Assignee: Manju Kalavala Currently NiFi publishes prometheus metrics which include bulletin gauges that reflects the existence of a specific bulletin. This request is to enhance the available metrics to include a counter metric that tracks the incidence of bulletins by category (e.g. ERROR, WARN, INFO). This will allow systems consuming metrics to determine increase or decrease of bulletins overtime. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-7796) Add Prometheus metrics for total bytes received and bytes sent for components
[ https://issues.apache.org/jira/browse/NIFI-7796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7796: --- Resolution: Fixed Status: Resolved (was: Patch Available) PR Merged 10/06/2020 > Add Prometheus metrics for total bytes received and bytes sent for components > - > > Key: NIFI-7796 > URL: https://issues.apache.org/jira/browse/NIFI-7796 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Yolanda M. Davis >Assignee: Matt Burgess >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently metrics available via Prometheus metrics ndpoint or reporting tasks > include gauges for niti_amount_bytes_received and nifi_amount_bytes_sent. > However in order to perform rate calculations with prometheus it would be > valuable to also expose counters for bytes_received and bytes_sent metrics > that are comparable to the existing values for nifi_total_bytes_read and > nifi_total_bytes_written. > Expected values would be nifi_total_bytes_received and nifi_total_bytes_sent. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7868) Processor Stats don't always include correct values for number of FlowFiles/Bytes sent
[ https://issues.apache.org/jira/browse/NIFI-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7868: --- Status: Patch Available (was: Open) > Processor Stats don't always include correct values for number of > FlowFiles/Bytes sent > -- > > Key: NIFI-7868 > URL: https://issues.apache.org/jira/browse/NIFI-7868 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When a Processor emits a SEND provenance event, the framework keeps track of > this, and shows this as one of the stats in the stats history. However, when > a processor emits a SEND event with the 'force' flag set to true (which is > the default) such as PutFile, the values are not counted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7868) Processor Stats don't always include correct values for number of FlowFiles/Bytes sent
[ https://issues.apache.org/jira/browse/NIFI-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7868: --- Resolution: Fixed Status: Resolved (was: Patch Available) PR was merged 09/30/2020 > Processor Stats don't always include correct values for number of > FlowFiles/Bytes sent > -- > > Key: NIFI-7868 > URL: https://issues.apache.org/jira/browse/NIFI-7868 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When a Processor emits a SEND provenance event, the framework keeps track of > this, and shows this as one of the stats in the stats history. However, when > a processor emits a SEND event with the 'force' flag set to true (which is > the default) such as PutFile, the values are not counted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7796) Add counter metrics for bytes received and bytes sent for components
Yolanda M. Davis created NIFI-7796: -- Summary: Add counter metrics for bytes received and bytes sent for components Key: NIFI-7796 URL: https://issues.apache.org/jira/browse/NIFI-7796 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently metrics available via Prometheus metrics ndpoint or reporting tasks include gauges for niti_amount_bytes_received and nifi_amount_bytes_sent. However in order to perform rate calculations with prometheus it would be valuable to also expose counters for bytes_received and bytes_sent metrics that are comparable to the existing values for nifi_total_bytes_read and nifi_total_bytes_written. Expected values would be nifi_total_bytes_received and nifi_total_bytes_sent. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-7437:
---
Status: Patch Available (was: Open)
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.11.4, 1.10.0
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Assignee: Yolanda M. Davis
>Priority: Critical
> Labels: features, performance
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000-0
[jira] [Commented] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107168#comment-17107168
]
Yolanda M. Davis commented on NIFI-7437:
[~diarworld] Thanks for this info. I am also seeing this issue on a single
node but just wanted to ensure I have similar settings since I have isolated
the culprit and have a refactor that I am testing. When the PR is available
you are more than welcome to review/test it out.
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.10.0, 1.11.4
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Assignee: Yolanda M. Davis
>Priority: Critical
> Labels: features, performance
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Pre
[jira] [Commented] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106847#comment-17106847
]
Yolanda M. Davis commented on NIFI-7437:
[~diarworld] do you mind providing information on your heap settings for your
dev environment? Also are you using a clustered setup?
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.10.0, 1.11.4
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Assignee: Yolanda M. Davis
>Priority: Critical
> Labels: features, performance
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
[jira] [Assigned] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis reassigned NIFI-7437:
--
Assignee: Yolanda M. Davis
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.10.0, 1.11.4
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Assignee: Yolanda M. Davis
>Priority: Critical
> Labels: features, performance
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseBytes=-1
> 20
[jira] [Commented] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106687#comment-17106687
]
Yolanda M. Davis commented on NIFI-7437:
[~diarworld] An update for you I actually was able to recreate this issue even
with the fix form NIFI-7087. I will assign this to myself and research further
for a fix.
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.10.0, 1.11.4
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Priority: Critical
> Labels: features, performance
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
[jira] [Commented] (NIFI-7437) UI is slow when nifi.analytics.predict.enabled is true
[
https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106320#comment-17106320
]
Yolanda M. Davis commented on NIFI-7437:
[~diarworld] thanks for identifying this issue. There is actually a fix that
has been merged (however isn't in the latest release) that should address this
problem. See https://issues.apache.org/jira/browse/NIFI-7087 for details on
this.
> UI is slow when nifi.analytics.predict.enabled is true
> --
>
> Key: NIFI-7437
> URL: https://issues.apache.org/jira/browse/NIFI-7437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI, Extensions
>Affects Versions: 1.10.0, 1.11.4
> Environment: Java11, CentOS8
>Reporter: Dmitry Ibragimov
>Priority: Critical
> Labels: features, performance
>
> We faced with issue when nifi.analytics.predict.enabled is true after cluster
> upgrade to 1.11.4
> We have about 4000 processors in development enviroment, but most of them is
> in disabled state: 256 running, 1263 stopped, 2543 disabled
> After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure
> prediction feature and enable it in configuration:
> {code:java}
> nifi.analytics.predict.enabled=true
> nifi.analytics.predict.interval=3 mins
> nifi.analytics.query.interval=5 mins
> nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
> nifi.analytics.connection.model.score.name=rSquared
> nifi.analytics.connection.model.score.threshold=.90
> {code}
> And we faced with terrible UI performance degradataion. Root page opens in 20
> seconds instead of 200-500ms. About ~100 times slower. I've tesed it with
> different environments centos7/8, java8/11, clustered secured, clustered
> unsecured, standalone unsecured - all the same.
> In debug log for ThreadPoolRequestReplicator:
> {code:java}
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max =
> 20625, average = 20161.0 ms
> 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/flow/process-groups/root (Request ID
> c144196f-d4cb-4053-8828-70e06f7c5100):
> newnifi01:8080: 19548 millis
> newnifi02:8080: 20625 millis
> newnifi03:8080: 20310 millis{code}
> More deep debug:
>
> {code:java}
> 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21]
> org.eclipse.jetty.server.HttpChannel REQUEST for
> //newnifi01:8080/nifi-api/flow/process-groups/root on
> HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0}
> GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1
> Host: newnifi01:8080
> ...
> 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> back pressure by content size in bytes. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time
> to back pressure by object count. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> object count for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting
> content size in bytes for next interval. Returning -1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalPercentageUseCount=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextIntervalBytes=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: timeToBytesBackpressureMillis=-1
> 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21]
> o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id
> eb602b2a-016f-1000--2767192a: nextInt
[jira] [Created] (NIFI-7408) Add Connection Percent Used Count/Byte Metrics for Metrics Endpoints
Yolanda M. Davis created NIFI-7408: -- Summary: Add Connection Percent Used Count/Byte Metrics for Metrics Endpoints Key: NIFI-7408 URL: https://issues.apache.org/jira/browse/NIFI-7408 Project: Apache NiFi Issue Type: New Feature Components: Core Framework, Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently certain metrics are available to calculate percentUsed of a connection. This provides an additional metric with value already calculated for simplicity. This should be made available via the NiFi metrics api and the Prometheus Reporting Task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7379) Prometheus components should not use the same registries or metric objects
[ https://issues.apache.org/jira/browse/NIFI-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7379: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merged 04/28/2020 > Prometheus components should not use the same registries or metric objects > -- > > Key: NIFI-7379 > URL: https://issues.apache.org/jira/browse/NIFI-7379 > Project: Apache NiFi > Issue Type: Bug >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently all Prometheus components in NiFi (the REST endpoint, the reporting > task, and the record sink) use the same set of metric objects and collection > registries. This can cause undesired behavior, such as causing label > conflicts (for different Instance Identifier values for example), undesired > metrics to be present (if QueryNiFiReportingTask adds metrics, > PrometheusReportingTask will expose them too), injection of bad data points > (if you have a bad query that overwrites an existing metric), etc. > Each component should have its own copy of the collection registries and > metric objects so as not to interfere with those of other Prometheus > components. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7378) Prometheus components give error when null labels are provided
[ https://issues.apache.org/jira/browse/NIFI-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7378: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merged 4/22/2020 > Prometheus components give error when null labels are provided > -- > > Key: NIFI-7378 > URL: https://issues.apache.org/jira/browse/NIFI-7378 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > It is possible for some components that a field used as a label in the > metrics will have a null value. In that case the scraping fails and thus no > metrics are available. > The code should be defensive in the sense of setting a default value for all > null labels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7273) Add flow metrics REST endpoint with for Prometheus scraping
[ https://issues.apache.org/jira/browse/NIFI-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7273: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merged on 4/3/2020 > Add flow metrics REST endpoint with for Prometheus scraping > --- > > Key: NIFI-7273 > URL: https://issues.apache.org/jira/browse/NIFI-7273 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > NiFi has the ability to expose endpoints for Prometheus to scrape via the > PrometheusReportingTask (NIFI-4362) and via components that use the > PrometheusRecordSink controller service. However that involves adding > components to the overall flow, which requires their own configuration and > ends up generating their own metrics that contribute to rollup metrics and > queries. > This Jira proposes to add an endpoint to the NiFi REST API that exposes the > following metrics/information in Prometheus format for scraping: > - Root Process Group status (recursive to include all components) > - Connection Status Analytics (backpressure predictions, e.g.) > - JVM Metrics > - Bulletins (for use by AlertManager, not necessarily a metric per se) > Standard security/authorization principles apply, and it is proposed to offer > node-specific metrics rather than cluster-wide aggregates, as Prometheus can > then choose how to do the aggregates as necessary. > It may be prudent to refactor PrometheusMetricsUtil out into its own module, > for use by the various components in various modules (to now include the > framework). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7163) Create RulesEngine and RulesEngineProvider Interfaces
[ https://issues.apache.org/jira/browse/NIFI-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-7163: --- Status: Patch Available (was: In Progress) > Create RulesEngine and RulesEngineProvider Interfaces > - > > Key: NIFI-7163 > URL: https://issues.apache.org/jira/browse/NIFI-7163 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The existing RulesEngineService allows for centralized execution of rules, > but there may be cases where processors (or other calling components) want to > operate directly with an instance of an engine (to prevent a bottleneck with > the controller service) or would like access to the rules driving the engine. > To facilitate this a RulesEngine and RulesEngineProvider interface should be > created to support the following: > *RulesEngine:* > Firing rules with provided facts and returning required actions > Checking rules with provided facts and return information on which rules > were fired from the available list > *RulesEngineProvider:* > Return an instance of a rules engine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7163) Create RulesEngine and RulesEngineProvider Interfaces
Yolanda M. Davis created NIFI-7163: -- Summary: Create RulesEngine and RulesEngineProvider Interfaces Key: NIFI-7163 URL: https://issues.apache.org/jira/browse/NIFI-7163 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis The existing RulesEngineService allows for centralized execution of rules, but there may be cases where processors (or other calling components) want to operate directly with an instance of an engine (to prevent a bottleneck with the controller service) or would like access to the rules driving the engine. To facilitate this a RulesEngine and RulesEngineProvider interface should be created to support the following: *RulesEngine:* Firing rules with provided facts and returning required actions Checking rules with provided facts and return information on which rules were fired from the available list *RulesEngineProvider:* Return an instance of a rules engine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7070) Enhance Action Handler Interface
Yolanda M. Davis created NIFI-7070: -- Summary: Enhance Action Handler Interface Key: NIFI-7070 URL: https://issues.apache.org/jira/browse/NIFI-7070 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently the ActionHandler interface allows NiFi flows to send Actions to other components for execution. In some cases that execution may need to return a value to the calling component (such as result information from a remote call or expression execution). The interface should be enhanced or extended to support returning some value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6962) Move HashService and HashAlgorithm to nifi-security-utils module
[ https://issues.apache.org/jira/browse/NIFI-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6962: --- Status: Patch Available (was: Open) > Move HashService and HashAlgorithm to nifi-security-utils module > > > Key: NIFI-6962 > URL: https://issues.apache.org/jira/browse/NIFI-6962 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HashService and HashAlgorithm class/enum have the potential to be leveraged > by other components to implement hashing functions, however they are only > packaged within the nifi-standard-processors module. Relocating to > nifi-security-utils would provide the appropriate context for these services > and give flexibility for implementation beyond the standard processor module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6962) Move HashService and HashAlgorithm to nifi-security-utils module
Yolanda M. Davis created NIFI-6962: -- Summary: Move HashService and HashAlgorithm to nifi-security-utils module Key: NIFI-6962 URL: https://issues.apache.org/jira/browse/NIFI-6962 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis HashService and HashAlgorithm class/enum have the potential to be leveraged by other components to implement hashing functions, however they are only packaged within the nifi-standard-processors module. Relocating to nifi-security-utils would provide the appropriate context for these services and give flexibility for implementation beyond the standard processor module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-6889) Create a Rules Processor
[ https://issues.apache.org/jira/browse/NIFI-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6889. Resolution: Won't Do Not sure if it's a generic enough fit for community contribution > Create a Rules Processor > > > Key: NIFI-6889 > URL: https://issues.apache.org/jira/browse/NIFI-6889 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Time Spent: 2.5h > Remaining Estimate: 0h > > Create a processor that supports sending flow file content and attributes for > evaluation by a rules engine. The resultant actions should be serialized as > flow files that can sent downstream. Processor should leverage the Rules > Service Controller Service and existing Rules API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6889) Create a Rules Processor
[ https://issues.apache.org/jira/browse/NIFI-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6889: --- Status: Open (was: Patch Available) > Create a Rules Processor > > > Key: NIFI-6889 > URL: https://issues.apache.org/jira/browse/NIFI-6889 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Time Spent: 2.5h > Remaining Estimate: 0h > > Create a processor that supports sending flow file content and attributes for > evaluation by a rules engine. The resultant actions should be serialized as > flow files that can sent downstream. Processor should leverage the Rules > Service Controller Service and existing Rules API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6889) Create a Rules Processor
[ https://issues.apache.org/jira/browse/NIFI-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6889: --- Status: Patch Available (was: In Progress) > Create a Rules Processor > > > Key: NIFI-6889 > URL: https://issues.apache.org/jira/browse/NIFI-6889 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Time Spent: 10m > Remaining Estimate: 0h > > Create a processor that supports sending flow file content and attributes for > evaluation by a rules engine. The resultant actions should be serialized as > flow files that can sent downstream. Processor should leverage the Rules > Service Controller Service and existing Rules API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6859) Add scripted versions of RecordSinkService, RulesEngineService, and ActionHandler
[ https://issues.apache.org/jira/browse/NIFI-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6859: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add scripted versions of RecordSinkService, RulesEngineService, and > ActionHandler > - > > Key: NIFI-6859 > URL: https://issues.apache.org/jira/browse/NIFI-6859 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Labels: rules > Time Spent: 50m > Remaining Estimate: 0h > > Now that the RecordSinkService, RulesEngineService, and ActionHandler > interfaces/APIs have been added to NiFi, it would be good to have scripted > implementations of these, in order to allow custom operations when these APIs > are invoked. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6859) Add scripted versions of RecordSinkService, RulesEngineService, and ActionHandler
[ https://issues.apache.org/jira/browse/NIFI-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6859: --- Labels: rules (was: ) > Add scripted versions of RecordSinkService, RulesEngineService, and > ActionHandler > - > > Key: NIFI-6859 > URL: https://issues.apache.org/jira/browse/NIFI-6859 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Labels: rules > Time Spent: 50m > Remaining Estimate: 0h > > Now that the RecordSinkService, RulesEngineService, and ActionHandler > interfaces/APIs have been added to NiFi, it would be good to have scripted > implementations of these, in order to allow custom operations when these APIs > are invoked. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6855) Add action type option to Action Handler
[ https://issues.apache.org/jira/browse/NIFI-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6855: --- Labels: rules (was: ) > Add action type option to Action Handler > > > Key: NIFI-6855 > URL: https://issues.apache.org/jira/browse/NIFI-6855 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Allow ActionHandlers to be configured to only execute for the configured > action type. This should be optional where if blank it will execute > regardless of type and as long as required attributes for the action are > provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6889) Create a Rules Processor
[ https://issues.apache.org/jira/browse/NIFI-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6889: --- Labels: rules (was: ) > Create a Rules Processor > > > Key: NIFI-6889 > URL: https://issues.apache.org/jira/browse/NIFI-6889 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > > Create a processor that supports sending flow file content and attributes for > evaluation by a rules engine. The resultant actions should be serialized as > flow files that can sent downstream. Processor should leverage the Rules > Service Controller Service and existing Rules API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6842) Create a Metrics Event Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6842: --- Labels: rules (was: ) > Create a Metrics Event Reporting Task > - > > Key: NIFI-6842 > URL: https://issues.apache.org/jira/browse/NIFI-6842 > Project: Apache NiFi > Issue Type: Sub-task > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Leveraging the Metrics Query, Rules Engine and Rules Action Handler APIs > create a Metrics Event Reporting Task which will allow NiFi to query for one > or more metric values, learn via rules if any action should be taken based on > retrieved values and report/send to the appropriate action handler. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6802) Rules-Driven NiFi Metrics Event Reporting
[ https://issues.apache.org/jira/browse/NIFI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6802: --- Fix Version/s: 1.11.0 > Rules-Driven NiFi Metrics Event Reporting > - > > Key: NIFI-6802 > URL: https://issues.apache.org/jira/browse/NIFI-6802 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > > Support the ability to query for specific NiFi metrics, evaluate those > metrics against a centralized set of rules (via a rules engine) and to > execute any required actions as dictated by those rules. For example users > could choose to query for connection related prediction metrics (such as time > to backpressure) send that information to a rules engine. If the rules engine > determine an action needs to be performed, such as producing a bulletin on > the UI or sending information downstream to an external system, those actions > would then be executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-6802) Rules-Driven NiFi Metrics Event Reporting
[ https://issues.apache.org/jira/browse/NIFI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6802. Resolution: Fixed > Rules-Driven NiFi Metrics Event Reporting > - > > Key: NIFI-6802 > URL: https://issues.apache.org/jira/browse/NIFI-6802 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > > Support the ability to query for specific NiFi metrics, evaluate those > metrics against a centralized set of rules (via a rules engine) and to > execute any required actions as dictated by those rules. For example users > could choose to query for connection related prediction metrics (such as time > to backpressure) send that information to a rules engine. If the rules engine > determine an action needs to be performed, such as producing a bulletin on > the UI or sending information downstream to an external system, those actions > would then be executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6803) Create a Rules Action Handler Service API and general implementation
[ https://issues.apache.org/jira/browse/NIFI-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6803: --- Labels: rules (was: ) > Create a Rules Action Handler Service API and general implementation > > > Key: NIFI-6803 > URL: https://issues.apache.org/jira/browse/NIFI-6803 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently the Rules API and Controller Service allows Reporting Task to > access centralized rules information and learn what actions should be > performed given a certain set of facts. However it currently leaves it to a > calling processor or controller service to interpret and execute the returned > actions. For this feature create an Action handler API and generalized set > of Action handlers which can be leveraged for the following: > Alerting: Display bulletins > Logging: Write to logs > Send/Report: Send information to a downstream component or external system > Expression Handling: Execute expressions directly given the type (e.g. MVEL, > SpEL, NiFi) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6802) Rules-Driven NiFi Metrics Event Reporting
[ https://issues.apache.org/jira/browse/NIFI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6802: --- Labels: rules (was: ) > Rules-Driven NiFi Metrics Event Reporting > - > > Key: NIFI-6802 > URL: https://issues.apache.org/jira/browse/NIFI-6802 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > > Support the ability to query for specific NiFi metrics, evaluate those > metrics against a centralized set of rules (via a rules engine) and to > execute any required actions as dictated by those rules. For example users > could choose to query for connection related prediction metrics (such as time > to backpressure) send that information to a rules engine. If the rules engine > determine an action needs to be performed, such as producing a bulletin on > the UI or sending information downstream to an external system, those actions > would then be executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6890) Allow rules definition in Easy Rules Controller Service configuration
[ https://issues.apache.org/jira/browse/NIFI-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6890: --- Labels: rules (was: ) > Allow rules definition in Easy Rules Controller Service configuration > - > > Key: NIFI-6890 > URL: https://issues.apache.org/jira/browse/NIFI-6890 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently the EasyRulesEngineService only provides an option to load rule > from a rules file. An option should also be provided to allow rules to be > defined within a property on the config UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6854) Add option to ignore rule when fact value(s) is not available
[ https://issues.apache.org/jira/browse/NIFI-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6854: --- Labels: rules (was: ) > Add option to ignore rule when fact value(s) is not available > -- > > Key: NIFI-6854 > URL: https://issues.apache.org/jira/browse/NIFI-6854 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > With the current implementation of the RulesEngineService if a fact that is > required for a rule does not exist in the provided Map the service will throw > and exception. Add an option to allow rules to be skipped if the fact > required is not provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6778) Rules Engine Controller Service with Default Easy Rules Implementation
[ https://issues.apache.org/jira/browse/NIFI-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6778: --- Labels: rules (was: ) > Rules Engine Controller Service with Default Easy Rules Implementation > -- > > Key: NIFI-6778 > URL: https://issues.apache.org/jira/browse/NIFI-6778 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Labels: rules > Fix For: 1.10.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Create a rules engine controller service in NiFi which would allow > processors, reporting tasks and/or other controller services to interrogate a > centralized engine for actions required given a certain set of data (or > facts). The rules engine should leverage a rules file or database to fire > rules and either return the list of action or execute required actions. > For a default implementation the EasyRules engine can be used however an > interface should be offered to allow other implementations. An internal > Rule/Action API can also be created to provide a common representation for > rules and actions within NiFi. This would allow Actions to be received and > processed by callers or allow Rules to generated and fired dynamically if > needed. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6890) Allow rules definition in Easy Rules Controller Service configuration
[ https://issues.apache.org/jira/browse/NIFI-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6890: --- Status: Patch Available (was: Open) > Allow rules definition in Easy Rules Controller Service configuration > - > > Key: NIFI-6890 > URL: https://issues.apache.org/jira/browse/NIFI-6890 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the EasyRulesEngineService only provides an option to load rule > from a rules file. An option should also be provided to allow rules to be > defined within a property on the config UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6890) Allow rules definition in Easy Rules Controller Service configuration
Yolanda M. Davis created NIFI-6890: -- Summary: Allow rules definition in Easy Rules Controller Service configuration Key: NIFI-6890 URL: https://issues.apache.org/jira/browse/NIFI-6890 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently the EasyRulesEngineService only provides an option to load rule from a rules file. An option should also be provided to allow rules to be defined within a property on the config UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6889) Create a Rules Processor
Yolanda M. Davis created NIFI-6889: -- Summary: Create a Rules Processor Key: NIFI-6889 URL: https://issues.apache.org/jira/browse/NIFI-6889 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Create a processor that supports sending flow file content and attributes for evaluation by a rules engine. The resultant actions should be serialized as flow files that can sent downstream. Processor should leverage the Rules Service Controller Service and existing Rules API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6855) Add action type option to Action Handler
[ https://issues.apache.org/jira/browse/NIFI-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6855: --- Status: Patch Available (was: In Progress) > Add action type option to Action Handler > > > Key: NIFI-6855 > URL: https://issues.apache.org/jira/browse/NIFI-6855 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Allow ActionHandlers to be configured to only execute for the configured > action type. This should be optional where if blank it will execute > regardless of type and as long as required attributes for the action are > provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6854) Add option to ignore rule when fact value(s) is not available
[ https://issues.apache.org/jira/browse/NIFI-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6854: --- Status: Patch Available (was: In Progress) > Add option to ignore rule when fact value(s) is not available > -- > > Key: NIFI-6854 > URL: https://issues.apache.org/jira/browse/NIFI-6854 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > With the current implementation of the RulesEngineService if a fact that is > required for a rule does not exist in the provided Map the service will throw > and exception. Add an option to allow rules to be skipped if the fact > required is not provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6854) Add option to ignore rule when fact value(s) is not available
[ https://issues.apache.org/jira/browse/NIFI-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969608#comment-16969608 ] Yolanda M. Davis commented on NIFI-6854: Enhanced this further to ignore any errors related to compiling a condition (which includes missing facts, syntax, etc). > Add option to ignore rule when fact value(s) is not available > -- > > Key: NIFI-6854 > URL: https://issues.apache.org/jira/browse/NIFI-6854 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > With the current implementation of the RulesEngineService if a fact that is > required for a rule does not exist in the provided Map the service will throw > and exception. Add an option to allow rules to be skipped if the fact > required is not provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6855) Add action type option to Action Handler
Yolanda M. Davis created NIFI-6855: -- Summary: Add action type option to Action Handler Key: NIFI-6855 URL: https://issues.apache.org/jira/browse/NIFI-6855 Project: Apache NiFi Issue Type: Improvement Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Allow ActionHandlers to be configured to only execute for the configured action type. This should be optional where if blank it will execute regardless of type and as long as required attributes for the action are provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6854) Add option to ignore rule when fact value(s) is not available
Yolanda M. Davis created NIFI-6854: -- Summary: Add option to ignore rule when fact value(s) is not available Key: NIFI-6854 URL: https://issues.apache.org/jira/browse/NIFI-6854 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.10.0 Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis With the current implementation of the RulesEngineService if a fact that is required for a rule does not exist in the provided Map the service will throw and exception. Add an option to allow rules to be skipped if the fact required is not provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6842) Create a Metrics Event Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6842: --- Status: Patch Available (was: In Progress) > Create a Metrics Event Reporting Task > - > > Key: NIFI-6842 > URL: https://issues.apache.org/jira/browse/NIFI-6842 > Project: Apache NiFi > Issue Type: Sub-task > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Leveraging the Metrics Query, Rules Engine and Rules Action Handler APIs > create a Metrics Event Reporting Task which will allow NiFi to query for one > or more metric values, learn via rules if any action should be taken based on > retrieved values and report/send to the appropriate action handler. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6819) Add RecordSink implementation for Kafka publishing
[ https://issues.apache.org/jira/browse/NIFI-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6819: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add RecordSink implementation for Kafka publishing > -- > > Key: NIFI-6819 > URL: https://issues.apache.org/jira/browse/NIFI-6819 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > With the advent of NIFI-6780, a RecordSinkService controller service > interface is available and is used by components like QueryNiFiReportingTask > to decouple the collection of data from its destination or transmission > method. Initial implementations in NIFI-6780 included a Site-to-Site record > sink as well as a DatabaseRecordSink for writing to RDBMS targets, and > NIFI-6799 added a PrometheusRecordSink. > This Jira proposes to add a RecordSinkService implementation for publishing > to Kafka. A RecordSetWriter property should be provided for formatting the > data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6842) Create a Metrics Event Reporting Task
Yolanda M. Davis created NIFI-6842: -- Summary: Create a Metrics Event Reporting Task Key: NIFI-6842 URL: https://issues.apache.org/jira/browse/NIFI-6842 Project: Apache NiFi Issue Type: Sub-task Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Fix For: 1.11.0 Leveraging the Metrics Query, Rules Engine and Rules Action Handler APIs create a Metrics Event Reporting Task which will allow NiFi to query for one or more metric values, learn via rules if any action should be taken based on retrieved values and report/send to the appropriate action handler. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6804) Create and Rules Action Handler Lookup Service
[ https://issues.apache.org/jira/browse/NIFI-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6804: --- Fix Version/s: 1.11.0 > Create and Rules Action Handler Lookup Service > -- > > Key: NIFI-6804 > URL: https://issues.apache.org/jira/browse/NIFI-6804 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Fix For: 1.11.0 > > > Create a lookup service that will allow dynamic lookup of Action Handlers > based on Action type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-6804) Create and Rules Action Handler Lookup Service
[ https://issues.apache.org/jira/browse/NIFI-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6804. Resolution: Fixed This was actually resolved within NIFI-6403 > Create and Rules Action Handler Lookup Service > -- > > Key: NIFI-6804 > URL: https://issues.apache.org/jira/browse/NIFI-6804 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > Create a lookup service that will allow dynamic lookup of Action Handlers > based on Action type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6803) Create a Rules Action Handler Service API and general implementation
[ https://issues.apache.org/jira/browse/NIFI-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6803: --- Status: Patch Available (was: Open) > Create a Rules Action Handler Service API and general implementation > > > Key: NIFI-6803 > URL: https://issues.apache.org/jira/browse/NIFI-6803 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the Rules API and Controller Service allows Reporting Task to > access centralized rules information and learn what actions should be > performed given a certain set of facts. However it currently leaves it to a > calling processor or controller service to interpret and execute the returned > actions. For this feature create an Action handler API and generalized set > of Action handlers which can be leveraged for the following: > Alerting: Display bulletins > Logging: Write to logs > Send/Report: Send information to a downstream component or external system > Expression Handling: Execute expressions directly given the type (e.g. MVEL, > SpEL, NiFi) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6801) Connection analytics models should be distinct for connections
[
https://issues.apache.org/jira/browse/NIFI-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6801:
---
Description:
Connections should have Status Analytics Model instances that are unique for
each connection such that calculations for the model can be referred to or
updated when new observations are available. Currently the model appears to be
shared across connection objects which leads to erroneous results, especially
when a connection does not have enough observations to update the model (since
the older values of a previous connection may still be present).
This behavior can be seen when debugging is enabled for analytics and noting
differences between a Timer Driven Thread execution of predictions vs the NiFi
Web Server thread predictions for multiple connections (a flow with one
connection would not exhibit this issue). The Time Driven thread may not
retrieve enough connection status observations to refresh the model, and when
that occurs, it may still contain calculations from a previous model and use
that for predictions. Observers may see behavior where web server and timer
driven predictions only have similar or matching predictions for one connection
but others may be mismatched, such as in the below (where connection has
similar prediction between threads but , and do not):
{code:java}
2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 00:44:13 / 2653416
2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 01:09:13 / 4153511
2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 00:32:31 / 1951609
2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 00:32:28 / 1948504
2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 07:29:14 / 26954837
2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 05:24:56 / 19496954
2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 04:08:36 / 14916150
2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 04:17:49 / 15469243
2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 01:13:42 / 4422829
2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 00:57:15 / 3435109
2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
backpressure = 07:28:46 / 26926608
2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
backpressure = 05:24:28 / 19468725
{code}
A workaround for this is to increase the nifi.analytics.query.interval
property or decrease the nifi.components.status.snapshot.frequency property to
ensure obtaining enough observations for a refresh. However an appropriate fix
is to ensure model instances are unique for each connection and to ensure that,
by default, all threads will obtain enough observations for predictions.
was:
Connections should have Status Analytics Model instances that are unique for
each connection such that calculations for the model can be referred to or
updated when new observations are available. Currently the model appears to be
shared across connection objects which leads to erroneous results, especially
when a connection does not have enough observations to update the model (since
the older values of a previous connection may still be present).
This behavior can be seen when debugging is enabled for analytics and noting
differences between a Timer Driven Thread execution of predictions vs the NiFi
Web Server thread predictions for multiple connections (a flow with one
connection would not exhibit this issue). The Time Driven thread may not
retrieve enough connection status observations to refresh the model, and when
that occurs, it may still contain calculations from a previous model and use
that for predictions. Observers may see behavior where web server and timer
driven predictions only have similar or matching predictions for one connection
but others may be mismatched, such as in the below (where con
[jira] [Updated] (NIFI-6801) Connection analytics models should be distinct for connections
[
https://issues.apache.org/jira/browse/NIFI-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6801:
---
Summary: Connection analytics models should be distinct for connections
(was: Connection analytics model should be unique across connections)
> Connection analytics models should be distinct for connections
> --
>
> Key: NIFI-6801
> URL: https://issues.apache.org/jira/browse/NIFI-6801
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.10.0
>Reporter: Yolanda M. Davis
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Connections should have Status Analytics Model instances that are unique for
> each connection such that calculations for the model can be referred to or
> updated when new observations are available. Currently the model appears to
> be shared across connection objects which leads to erroneous results,
> especially when a connection does not have enough observations to update the
> model (since the older values of a previous connection may still be present).
> This behavior can be seen when debugging is enabled for analytics and noting
> differences between a Timer Driven Thread execution of predictions vs the
> NiFi Web Server thread predictions for multiple connections (a flow with one
> connection would not exhibit this issue). The Time Driven thread may not
> retrieve enough connection status observations to refresh the model, and when
> that occurs, it may still contain calculations from a previous model and use
> that for predictions. Observers may see behavior where web server and timer
> driven predictions only have similar or matching predictions for one
> connection but others may be mismatched, such as in the below (where
> connection has similar prediction between threads but , and do
> not):
> {code:java}
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 00:44:13 / 2653416
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 01:09:13 / 4153511
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 00:32:31 / 1951609
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 00:32:28 / 1948504
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 07:29:14 / 26954837
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 05:24:56 / 19496954
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 04:08:36 / 14916150
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 04:17:49 / 15469243
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 01:13:42 / 4422829
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 00:57:15 / 3435109
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 07:28:46 / 26926608
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 05:24:28 / 19468725
> {code}
> A workaround for this is to increase the nifi.analytics.query.interval
> property or decrease the nifi.components.status.snapshot.frequency property
> to ensure obtaining enough observations for a refresh. However an appropriate
> fix is to ensure model instances are unique for each connection and to ensure
> that by default timer driven threads will obtain enough observations for
> predictions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-6804) Create and Rules Action Handler Lookup Service
Yolanda M. Davis created NIFI-6804: -- Summary: Create and Rules Action Handler Lookup Service Key: NIFI-6804 URL: https://issues.apache.org/jira/browse/NIFI-6804 Project: Apache NiFi Issue Type: Sub-task Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Create a lookup service that will allow dynamic lookup of Action Handlers based on Action type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6803) Create a Rules Action Handler Service API and general implementation
Yolanda M. Davis created NIFI-6803: -- Summary: Create a Rules Action Handler Service API and general implementation Key: NIFI-6803 URL: https://issues.apache.org/jira/browse/NIFI-6803 Project: Apache NiFi Issue Type: Sub-task Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently the Rules API and Controller Service allows Reporting Task to access centralized rules information and learn what actions should be performed given a certain set of facts. However it currently leaves it to a calling processor or controller service to interpret and execute the returned actions. For this feature create an Action handler API and generalized set of Action handlers which can be leveraged for the following: Alerting: Display bulletins Logging: Write to logs Send/Report: Send information to a downstream component or external system Expression Handling: Execute expressions directly given the type (e.g. MVEL, SpEL, NiFi) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6802) Rules-Driven NiFi Metrics Event Reporting
Yolanda M. Davis created NIFI-6802: -- Summary: Rules-Driven NiFi Metrics Event Reporting Key: NIFI-6802 URL: https://issues.apache.org/jira/browse/NIFI-6802 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Support the ability to query for specific NiFi metrics, evaluate those metrics against a centralized set of rules (via a rules engine) and to execute any required actions as dictated by those rules. For example users could choose to query for connection related prediction metrics (such as time to backpressure) send that information to a rules engine. If the rules engine determine an action needs to be performed, such as producing a bulletin on the UI or sending information downstream to an external system, those actions would then be executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6801) Connection analytics model should be unique across connections
[
https://issues.apache.org/jira/browse/NIFI-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6801:
---
Status: Patch Available (was: In Progress)
> Connection analytics model should be unique across connections
> --
>
> Key: NIFI-6801
> URL: https://issues.apache.org/jira/browse/NIFI-6801
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.10.0
>Reporter: Yolanda M. Davis
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Connections should have Status Analytics Model instances that are unique for
> each connection such that calculations for the model can be referred to or
> updated when new observations are available. Currently the model appears to
> be shared across connection objects which leads to erroneous results,
> especially when a connection does not have enough observations to update the
> model (since the older values of a previous connection may still be present).
> This behavior can be seen when debugging is enabled for analytics and noting
> differences between a Timer Driven Thread execution of predictions vs the
> NiFi Web Server thread predictions for multiple connections (a flow with one
> connection would not exhibit this issue). The Time Driven thread may not
> retrieve enough connection status observations to refresh the model, and when
> that occurs, it may still contain calculations from a previous model and use
> that for predictions. Observers may see behavior where web server and timer
> driven predictions only have similar or matching predictions for one
> connection but others may be mismatched, such as in the below (where
> connection has similar prediction between threads but , and do
> not):
> {code:java}
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 00:44:13 / 2653416
> 2019-10-22 08:01:34,539 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 01:09:13 / 4153511
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 00:32:31 / 1951609
> 2019-10-22 08:01:34,540 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 00:32:28 / 1948504
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 07:29:14 / 26954837
> 2019-10-22 08:01:34,541 INFO [NiFi Web Server-23]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 05:24:56 / 19496954
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 04:08:36 / 14916150
> 2019-10-22 08:02:02,767 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 04:17:49 / 15469243
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 01:13:42 / 4422829
> 2019-10-22 08:02:02,769 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 00:57:15 / 3435109
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to COUNT
> backpressure = 07:28:46 / 26926608
> 2019-10-22 08:02:02,770 INFO [Timer-Driven Process Thread-2]
> o.a.nifi.reporting.StandardEventAccess : predicted time to BYTES
> backpressure = 05:24:28 / 19468725
> {code}
> A workaround for this is to increase the nifi.analytics.query.interval
> property or decrease the nifi.components.status.snapshot.frequency property
> to ensure obtaining enough observations for a refresh. However an appropriate
> fix is to ensure model instances are unique for each connection and to ensure
> that by default timer driven threads will obtain enough observations for
> predictions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-6780) Create a Metrics Query Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6780: --- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Create a Metrics Query Reporting Task > - > > Key: NIFI-6780 > URL: https://issues.apache.org/jira/browse/NIFI-6780 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Yolanda M. Davis >Assignee: Matt Burgess >Priority: Major > Fix For: 1.10.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently NiFi has metrics reporting tasks which have a specific set of > metrics that are sent out to via site-to-site or other protocols. To expand > upon this a query based reporting task is proposed to provide users the > flexibility to select the types of metrics and the conditions on they should > be reported using sql like statements. > It may be desired that the results of a query are transmitted to any number > of targets. The current pattern is to implement this as a Site-to-Site > reporting task, but that puts the onus on the user to create a sub-flow with > an Input Port for receiving the S2S messages, and it creates additional > provenance events for these. A new approach to be considered here is to > decouple the results from the destination. Proposed is a RecordSinkService > controller service interface, which the query-based reporting task uses to > transmit the query results. The configured RecordSinkService implementation > would be responsible for the actual transmission of results to the sink. > Possible initial implementations include a Site-To-Site RecordSink (for > feature parity with the other reporting tasks) and a DatabaseRecordSink (to > transmit the query results to an external RDBMS using DBCPConnectionPool). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6778) Rules Engine Controller Service with Default Easy Rules Implementation
[ https://issues.apache.org/jira/browse/NIFI-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6778: --- Status: Patch Available (was: In Progress) > Rules Engine Controller Service with Default Easy Rules Implementation > -- > > Key: NIFI-6778 > URL: https://issues.apache.org/jira/browse/NIFI-6778 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Create a rules engine controller service in NiFi which would allow > processors, reporting tasks and/or other controller services to interrogate a > centralized engine for actions required given a certain set of data (or > facts). The rules engine should leverage a rules file or database to fire > rules and either return the list of action or execute required actions. > For a default implementation the EasyRules engine can be used however an > interface should be offered to allow other implementations. An internal > Rule/Action API can also be created to provide a common representation for > rules and actions within NiFi. This would allow Actions to be received and > processed by callers or allow Rules to generated and fired dynamically if > needed. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6780) Create a Metrics Query Reporting Task
Yolanda M. Davis created NIFI-6780: -- Summary: Create a Metrics Query Reporting Task Key: NIFI-6780 URL: https://issues.apache.org/jira/browse/NIFI-6780 Project: Apache NiFi Issue Type: New Feature Reporter: Yolanda M. Davis Assignee: Matt Burgess Currently NiFi has metrics reporting tasks which have a specific set of metrics that are sent out to via site-to-site or other protocols. To expand upon this a query based reporting task is proposed to provide users the flexibility to select the types of metrics and the conditions on they should be reported using sql like statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6778) Rules Engine Controller Service with Default Easy Rules Implementation
[ https://issues.apache.org/jira/browse/NIFI-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6778: --- Description: Create a rules engine controller service in NiFi which would allow processors, reporting tasks and/or other controller services to interrogate a centralized engine for actions required given a certain set of data (or facts). The rules engine should leverage a rules file or database to fire rules and either return the list of action or execute required actions. For a default implementation the EasyRules engine can be used however an interface should be offered to allow other implementations. An internal Rule/Action API can also be created to provide a common representation for rules and actions within NiFi. This would allow Actions to be received and processed by callers or allow Rules to generated and fired dynamically if needed. was: Create a rules engine controller service in NiFi which would allow processors, reporting tasks and/or other controller services to interrogate a centralized engine for actions required given a certain set of data (or facts). The rules engine should leverage a rules file or database to fire rules and either return the list of action or execute required actions. For a default implementation the EasyRules engine can be used however an interface should be offered to allow other implementations. An internal Rule/Action API can also be created to provide a common representation for rules and actions within NiFi. This would allow Actions to be received and processed by callers or allow Rules to generated and fired dynamically if needed. The interface should also support return actions that should be performed or execution of actions by the engine. > Rules Engine Controller Service with Default Easy Rules Implementation > -- > > Key: NIFI-6778 > URL: https://issues.apache.org/jira/browse/NIFI-6778 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > Create a rules engine controller service in NiFi which would allow > processors, reporting tasks and/or other controller services to interrogate a > centralized engine for actions required given a certain set of data (or > facts). The rules engine should leverage a rules file or database to fire > rules and either return the list of action or execute required actions. > For a default implementation the EasyRules engine can be used however an > interface should be offered to allow other implementations. An internal > Rule/Action API can also be created to provide a common representation for > rules and actions within NiFi. This would allow Actions to be received and > processed by callers or allow Rules to generated and fired dynamically if > needed. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-6778) Rules Engine Controller Service with Default Easy Rules Implementation
Yolanda M. Davis created NIFI-6778: -- Summary: Rules Engine Controller Service with Default Easy Rules Implementation Key: NIFI-6778 URL: https://issues.apache.org/jira/browse/NIFI-6778 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Create a rules engine controller service in NiFi which would allow processors, reporting tasks and/or other controller services to interrogate a centralized engine for actions required given a certain set of data (or facts). The rules engine should leverage a rules file or database to fire rules and either return the list of action or execute required actions. For a default implementation the EasyRules engine can be used however an interface should be offered to allow other implementations. An internal Rule/Action API can also be created to provide a common representation for rules and actions within NiFi. This would allow Actions to be received and processed by callers or allow Rules to generated and fired dynamically if needed. The interface should also support return actions that should be performed or execution of actions by the engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6649) Back Pressure Prediction: Separate query interval configuration from prediction interval
[ https://issues.apache.org/jira/browse/NIFI-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6649: --- Fix Version/s: 1.10.0 > Back Pressure Prediction: Separate query interval configuration from > prediction interval > > > Key: NIFI-6649 > URL: https://issues.apache.org/jira/browse/NIFI-6649 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently the interval setting `nifi.analytics.predict.interval` dictates > both how observations should be queried as well as how far to look into the > future. These should be separated to allow users to project further into the > future using few past predictions (or vice versa). For example allowing to > predict 60 minutes into the future using the last 5 minutes of data. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6649) Back Pressure Prediction: Separate query interval configuration from prediction interval
[ https://issues.apache.org/jira/browse/NIFI-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6649: --- Status: Patch Available (was: In Progress) > Back Pressure Prediction: Separate query interval configuration from > prediction interval > > > Key: NIFI-6649 > URL: https://issues.apache.org/jira/browse/NIFI-6649 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the interval setting `nifi.analytics.predict.interval` dictates > both how observations should be queried as well as how far to look into the > future. These should be separated to allow users to project further into the > future using few past predictions (or vice versa). For example allowing to > predict 60 minutes into the future using the last 5 minutes of data. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6649) Back Pressure Prediction: Separate query interval configuration from prediction interval
Yolanda M. Davis created NIFI-6649: -- Summary: Back Pressure Prediction: Separate query interval configuration from prediction interval Key: NIFI-6649 URL: https://issues.apache.org/jira/browse/NIFI-6649 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.10.0 Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently the interval setting `nifi.analytics.predict.interval` dictates both how observations should be queried as well as how far to look into the future. These should be separated to allow users to project further into the future using few past predictions (or vice versa). For example allowing to predict 60 minutes into the future using the last 5 minutes of data. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Issue Type: New Feature (was: Improvement)
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
This feature was merged 9/9/2019
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (NIFI-6574) Add check for valid predictionIntervalMillis value
[ https://issues.apache.org/jira/browse/NIFI-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6574. Fix Version/s: 1.10.0 Resolution: Fixed This appears to have been resolved in the context of NIFI-6510 parent PR commits. Setting to resolved > Add check for valid predictionIntervalMillis value > -- > > Key: NIFI-6574 > URL: https://issues.apache.org/jira/browse/NIFI-6574 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Priority: Minor > Fix For: 1.10.0 > > > For backpressure prediction the predictionIntervalMillis should have a > validity check for value provided. If invalid default value should be set > with a warning in logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6511) Specify remote protocol for predictive analytics
[ https://issues.apache.org/jira/browse/NIFI-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6511: --- Parent: (was: NIFI-6510) Issue Type: New Feature (was: Sub-task) > Specify remote protocol for predictive analytics > > > Key: NIFI-6511 > URL: https://issues.apache.org/jira/browse/NIFI-6511 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Andrew Christianson >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In order to allow more resource-demanding analytics to be run outside of the > main NiFi process, we need to define a protocol which allows NiFi to send > analytic input data (e.g. NiFi metrics) and receive > predictions/answers/useful values back. > Because the amount of input data to models may be large, the protocol needs > to be lightweight and efficient. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Assignee: Yolanda M. Davis
Status: Patch Available (was: Open)
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
---
Fix Version/s: 1.10.0
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6574) Add check for valid predictionIntervalMillis value
[ https://issues.apache.org/jira/browse/NIFI-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6574: --- Description: For backpressure prediction the predictionIntervalMillis should have a validity check for value provided. If invalid default value should be set with a warning in logs. (was: For the predictionIntervalMillis, add a validity check for value provided. If invalid default value should be set with an warning in logs.) > Add check for valid predictionIntervalMillis value > -- > > Key: NIFI-6574 > URL: https://issues.apache.org/jira/browse/NIFI-6574 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Priority: Minor > > For backpressure prediction the predictionIntervalMillis should have a > validity check for value provided. If invalid default value should be set > with a warning in logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6585) Update tests to decouple model validation from status analytics & engine validation
[ https://issues.apache.org/jira/browse/NIFI-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6585: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Update tests to decouple model validation from status analytics & engine > validation > --- > > Key: NIFI-6585 > URL: https://issues.apache.org/jira/browse/NIFI-6585 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The Connection Status Analytics object and engine test currently operates > more as an integration test between the model and those instances. We really > should mock the model and the extract functions (since separate tests exist > for that). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (NIFI-6586) Comments and Documentation for Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis resolved NIFI-6586. Resolution: Fixed > Comments and Documentation for Admin Guide > -- > > Key: NIFI-6586 > URL: https://issues.apache.org/jira/browse/NIFI-6586 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > -Update code with comments as needed. > -Include summary of analytics engine as well as configuration guide in admin > docs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6586) Comments and Documentation for Admin Guide
Yolanda M. Davis created NIFI-6586: -- Summary: Comments and Documentation for Admin Guide Key: NIFI-6586 URL: https://issues.apache.org/jira/browse/NIFI-6586 Project: Apache NiFi Issue Type: Sub-task Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis -Update code with comments as needed. -Include summary of analytics engine as well as configuration guide in admin docs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6585) Update tests to decouple model validation from status analytics & engine validation
[ https://issues.apache.org/jira/browse/NIFI-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6585: --- Status: Patch Available (was: In Progress) > Update tests to decouple model validation from status analytics & engine > validation > --- > > Key: NIFI-6585 > URL: https://issues.apache.org/jira/browse/NIFI-6585 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The Connection Status Analytics object and engine test currently operates > more as an integration test between the model and those instances. We really > should mock the model and the extract functions (since separate tests exist > for that). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6585) Update tests to decouple model validation from status analytics & engine validation
Yolanda M. Davis created NIFI-6585: -- Summary: Update tests to decouple model validation from status analytics & engine validation Key: NIFI-6585 URL: https://issues.apache.org/jira/browse/NIFI-6585 Project: Apache NiFi Issue Type: Sub-task Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis The Connection Status Analytics object and engine test currently operates more as an integration test between the model and those instances. We really should mock the model and the extract functions (since separate tests exist for that). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6566: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merged into analytics-framework by [~mattyb149] > Decouple status analytics model from connection status analytics objects > - > > Key: NIFI-6566 > URL: https://issues.apache.org/jira/browse/NIFI-6566 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Currently the connection status analytics object is tightly coupled with the > type of model used for prediction. This should be refactored to allow > flexibility for model selection. Users should also be able to configure the > model used via nifi.properties as described below: > > nifi.analytics.connection.model=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares > nifi.analytics.connection.model.score=rSquared > nifi.anlytics.connecttion.model.score.threshold=.90 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6566: --- Description: Currently the connection status analytics object is tightly coupled with the type of model used for prediction. This should be refactored to allow flexibility for model selection. Users should also be able to configure the model used via nifi.properties as described below: nifi.analytics.connection.model=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares nifi.analytics.connection.model.score=rSquared nifi.anlytics.connecttion.model.score.threshold=.90 was: Currently the connection status analytics object is tightly coupled with the type of model used for prediction. This should be refactored to allow flexibility for model selection. Users should also be able to configure the model used via nifi.properties as described below: nifi.analytics.connection.model=OrdinaryLeastSquaresMSAM nifi.analytics.connection.online=true nifi.analytics.connection.model.[param]= > Decouple status analytics model from connection status analytics objects > - > > Key: NIFI-6566 > URL: https://issues.apache.org/jira/browse/NIFI-6566 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the connection status analytics object is tightly coupled with the > type of model used for prediction. This should be refactored to allow > flexibility for model selection. Users should also be able to configure the > model used via nifi.properties as described below: > > nifi.analytics.connection.model=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares > nifi.analytics.connection.model.score=rSquared > nifi.anlytics.connecttion.model.score.threshold=.90 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6566) Decouple status analytics model from connection status analytics objects
[ https://issues.apache.org/jira/browse/NIFI-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6566: --- Status: Patch Available (was: In Progress) > Decouple status analytics model from connection status analytics objects > - > > Key: NIFI-6566 > URL: https://issues.apache.org/jira/browse/NIFI-6566 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently the connection status analytics object is tightly coupled with the > type of model used for prediction. This should be refactored to allow > flexibility for model selection. Users should also be able to configure the > model used via nifi.properties as described below: > > nifi.analytics.connection.model=OrdinaryLeastSquaresMSAM > nifi.analytics.connection.online=true > nifi.analytics.connection.model.[param]= > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6574) Add check for valid predictionIntervalMillis value
Yolanda M. Davis created NIFI-6574: -- Summary: Add check for valid predictionIntervalMillis value Key: NIFI-6574 URL: https://issues.apache.org/jira/browse/NIFI-6574 Project: Apache NiFi Issue Type: Sub-task Reporter: Yolanda M. Davis For the predictionIntervalMillis, add a validity check for value provided. If invalid default value should be set with an warning in logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6566) Decouple status analytics model from connection status analytics objects
Yolanda M. Davis created NIFI-6566: -- Summary: Decouple status analytics model from connection status analytics objects Key: NIFI-6566 URL: https://issues.apache.org/jira/browse/NIFI-6566 Project: Apache NiFi Issue Type: Sub-task Components: Core Framework Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis Currently the connection status analytics object is tightly coupled with the type of model used for prediction. This should be refactored to allow flexibility for model selection. Users should also be able to configure the model used via nifi.properties as described below: nifi.analytics.connection.model=OrdinaryLeastSquaresMSAM nifi.analytics.connection.online=true nifi.analytics.connection.model.[param]= -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6429: --- Status: Patch Available (was: In Progress) > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > SiteToSiteMetricsReportingTask > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-6429: --- Description: In some implementations of the SiteToSite Reporting Tasks there are variables captured which could have null values. In these cases those values are stripped out from the final output by default. Recommend adding a configurable option for users to opt to include null values to help ensure a consistent schema if needed (with default option being to remove values for backwards compatibility). Affected Reporting Tasks: SiteToSiteProvenanceReportingTask SiteToSiteStatusReportingTask SiteToSiteBulletinReportingTask SiteToSiteMetricsReportingTask was: In some implementations of the SiteToSite Reporting Tasks there are variables captured which could have null values. In these cases those values are stripped out from the final output by default. Recommend adding a configurable option for users to opt to include null values to help ensure a consistent schema if needed (with default option being to remove values for backwards compatibility). Affected Reporting Tasks: SiteToSiteProvenanceReportingTask SiteToSiteStatusReportingTask SiteToSiteBulletinReportingTask > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > SiteToSiteMetricsReportingTask > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis reassigned NIFI-6429: -- Assignee: Yolanda M. Davis > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6417) JOLT processors should accept more character encodings
[ https://issues.apache.org/jira/browse/NIFI-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885810#comment-16885810 ] Yolanda M. Davis commented on NIFI-6417: Also was chatting with [~mattyb149] offline and he mentioned that JoltTransformRecord is hardcoding things to UTF-8 behind the scenes even for input and that changes for this Jira should only target JolTransformJSON. [~mattyb149] please feel free to provide more details. > JOLT processors should accept more character encodings > -- > > Key: NIFI-6417 > URL: https://issues.apache.org/jira/browse/NIFI-6417 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Travis Neeley >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently JoltTransformJSON and JoltTransformRecord are hard-coded to input > and output UTF-8 encoding. These processors should be able to at least accept > a user-configured input encoding such as UTF-16, ISO-8859-1, etc. > We could retain the behavior of writing out in UTF-8, but I think for > consistency and flexibility we should add an Output Character Encoding > property as well. Both would be defaulted to UTF-8 to maintain current > default behavior. > *NOTE*: There may be a limitation at present with JoltTransformRecord > regarding String fields, where they are hard-coded to be UTF-8. If that is > the case, then both processors should still offer the Input Character > Encoding, but only JoltTransformJSON would have the Output Character Encoding > property. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (NIFI-6417) JOLT processors should accept more character encodings
[ https://issues.apache.org/jira/browse/NIFI-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885805#comment-16885805 ] Yolanda M. Davis edited comment on NIFI-6417 at 7/16/19 3:33 AM: - [~tneeley] just wanted to add some details concerning the Advanced UI. It is written in Angular *Front End* [transformjson.view.html|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.view.html#L32-L37] - This file defines the HTML view/layout for the Advanced View. Two dropdowns should be added for the input/output character sets with models (option values) populated in the controller (see below). [transformjson.controller.js|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L77-L83] - This file serves as the controller or glue to bind view to data and services. Here we'll need to retrieve the options that will populate the charset values in the view (they should already be part of the existing processor details call) . Also the existing [joltSpec javascript object |https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L261-L271]would need to be enhanced to send back the selected input and output encoding selected in the UI. *Back End:* [JoltSpecificationDTO|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/dto/JoltSpecificationDTO.java] - This is the data transfer object that is represented by jsonSpec in transformjson.controller.js. It needs be changed to represent the new values added for input/output charsets [TransformJSONResource|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java] - This endpoint is what actually run the test transformation.. The change here would be to use the values passed in via the JoltSpecificationDTO instead of the hardcoded DEFAULT_CHARSET value. One early recommendation is to start on the back end first making changes and using the TestTransformJSONResource class to confirm, before moving to the front end (since it can be tedious to test). was (Author: yolandamdavis): [~tneeley] just wanted to add some details concerning the Advanced UI. It is written in Angular *Front End* [transformjson.view.html|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.view.html#L32-L37] - This file defines the HTML view/layout for the Advanced View. Two dropdowns should be added for the input/output character sets with models (option values) populated in the controller (see below). [transformjson.controller.js|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L77-L83]] - This file serves as the controller or glue to bind view to data and services. Here we'll need to retrieve the options that will populate the charset values in the view (they should already be part of the existing processor details call) . Also the existing [joltSpec javascript object |[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L261-L271]]would need to be enhanced to send back the selected input and output encoding selected in the UI. *Back End:* [JoltSpecificationDTO|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/dto/JoltSpecificationDTO.java]] - This is the data transfer object that is represented by jsonSpec in transformjson.controller.js. It needs be changed to represent the new values added for input/output charsets [TransformJSONResource|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java]] - This endpoint is w
[jira] [Comment Edited] (NIFI-6417) JOLT processors should accept more character encodings
[ https://issues.apache.org/jira/browse/NIFI-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885805#comment-16885805 ] Yolanda M. Davis edited comment on NIFI-6417 at 7/16/19 3:30 AM: - [~tneeley] just wanted to add some details concerning the Advanced UI. It is written in Angular *Front End* [transformjson.view.html|https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.view.html#L32-L37] - This file defines the HTML view/layout for the Advanced View. Two dropdowns should be added for the input/output character sets with models (option values) populated in the controller (see below). [transformjson.controller.js|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L77-L83]] - This file serves as the controller or glue to bind view to data and services. Here we'll need to retrieve the options that will populate the charset values in the view (they should already be part of the existing processor details call) . Also the existing [joltSpec javascript object |[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L261-L271]]would need to be enhanced to send back the selected input and output encoding selected in the UI. *Back End:* [JoltSpecificationDTO|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/dto/JoltSpecificationDTO.java]] - This is the data transfer object that is represented by jsonSpec in transformjson.controller.js. It needs be changed to represent the new values added for input/output charsets [TransformJSONResource|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java]] - This endpoint is what actually run the test transformation.. The change here would be to use the values passed in via the JoltSpecificationDTO instead of the hardcoded DEFAULT_CHARSET value. One early recommendation is to start on the back end first making changes and using the TestTransformJSONResource class to confirm, before moving to the front end (since it can be tedious to test). was (Author: yolandamdavis): [~tneeley] just wanted to add some details concerning the Advanced UI. It is written in Angular *Front End* [transformjson.view.html|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.view.html#L32-L37]] - This file defines the HTML view/layout for the Advanced View. Two dropdowns should be added for the input/output character sets with models (option values) populated in the controller (see below). [transformjson.controller.js|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L77-L83]] - This file serves as the controller or glue to bind view to data and services. Here we'll need to retrieve the options that will populate the charset values in the view (they should already be part of the existing processor details call) . Also the existing [joltSpec javascript object |[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L261-L271]]would need to be enhanced to send back the selected input and output encoding selected in the UI. *Back End:* [JoltSpecificationDTO|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/dto/JoltSpecificationDTO.java]] - This is the data transfer object that is represented by jsonSpec in transformjson.controller.js. It needs be changed to represent the new values added for input/output charsets [TransformJSONResource|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java]] - This en
[jira] [Commented] (NIFI-6417) JOLT processors should accept more character encodings
[ https://issues.apache.org/jira/browse/NIFI-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885805#comment-16885805 ] Yolanda M. Davis commented on NIFI-6417: [~tneeley] just wanted to add some details concerning the Advanced UI. It is written in Angular *Front End* [transformjson.view.html|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.view.html#L32-L37]] - This file defines the HTML view/layout for the Advanced View. Two dropdowns should be added for the input/output character sets with models (option values) populated in the controller (see below). [transformjson.controller.js|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L77-L83]] - This file serves as the controller or glue to bind view to data and services. Here we'll need to retrieve the options that will populate the charset values in the view (they should already be part of the existing processor details call) . Also the existing [joltSpec javascript object |[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/webapp/app/transformjson/transformjson.controller.js#L261-L271]]would need to be enhanced to send back the selected input and output encoding selected in the UI. *Back End:* [JoltSpecificationDTO|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/dto/JoltSpecificationDTO.java]] - This is the data transfer object that is represented by jsonSpec in transformjson.controller.js. It needs be changed to represent the new values added for input/output charsets [TransformJSONResource|[https://github.com/apache/nifi/blob/c0185ef8ecf5490938bdac63b6d02575a5cd0808/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java]] - This endpoint is what actually run the test transformation.. The change here would be to use the values passed in via the JoltSpecificationDTO instead of the hardcoded DEFAULT_CHARSET value. One early recommendation is to start on the back end first making changes and using the TestTransformJSONResource class to confirm, before moving to the front end (since it can be tedious to test). > JOLT processors should accept more character encodings > -- > > Key: NIFI-6417 > URL: https://issues.apache.org/jira/browse/NIFI-6417 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Travis Neeley >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently JoltTransformJSON and JoltTransformRecord are hard-coded to input > and output UTF-8 encoding. These processors should be able to at least accept > a user-configured input encoding such as UTF-16, ISO-8859-1, etc. > We could retain the behavior of writing out in UTF-8, but I think for > consistency and flexibility we should add an Output Character Encoding > property as well. Both would be defaulted to UTF-8 to maintain current > default behavior. > *NOTE*: There may be a limitation at present with JoltTransformRecord > regarding String fields, where they are hard-coded to be UTF-8. If that is > the case, then both processors should still offer the Input Character > Encoding, but only JoltTransformJSON would have the Output Character Encoding > property. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
Yolanda M. Davis created NIFI-6429: -- Summary: Provide option to include null values for SiteToSite Reporting Tasks Key: NIFI-6429 URL: https://issues.apache.org/jira/browse/NIFI-6429 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.9.2 Reporter: Yolanda M. Davis In some implementations of the SiteToSite Reporting Tasks there are variables captured which could have null values. In these cases those values are stripped out from the final output by default. Recommend adding a configurable option for users to opt to include null values to help ensure a consistent schema if needed (with default option being to remove values for backwards compatibility). Affected Reporting Tasks: SiteToSiteProvenanceReportingTask SiteToSiteStatusReportingTask SiteToSiteBulletinReportingTask -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (NIFI-6254) TLS Toolkit in server mode does not indicate missing JDK JCE Unlimited Strength Support
[
https://issues.apache.org/jira/browse/NIFI-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6254:
---
Priority: Minor (was: Major)
> TLS Toolkit in server mode does not indicate missing JDK JCE Unlimited
> Strength Support
> ---
>
> Key: NIFI-6254
> URL: https://issues.apache.org/jira/browse/NIFI-6254
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security, Tools and Build
>Affects Versions: 1.9.2
> Environment: JDK 1.8.112
> Centos 7
>Reporter: Yolanda M. Davis
>Assignee: Andy LoPresto
>Priority: Minor
>
> When attempting to run the TLS Toolkit in server mode using a JDK that is not
> configured with the JCE Unlimited Strength policy files, the toolkit returns
> an error that does not clearly indicate the issue.
> For example when running the following command that uses a Java Home without:
> ${NIFI_TOOLKIT_DIST}/bin/tls-toolkit.sh server -F -f
> ${NIFI_TOOLKIT_CA_CONFIG_FILE}
> The scripts responds with the following:
> "Service server error: null"
> To surface this error it was necessary to do remote debugging (to see
> InvocationException errors were being thrown yet not shown). The request is
> to surface this type of error in order for users to take corrective action
> (in this case ensuring the JCE policy files are added to the Java SDK Home).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-6254) TLS Toolkit in server mode does not indicate missing JDK JCE Unlimited Strength Support
Yolanda M. Davis created NIFI-6254:
--
Summary: TLS Toolkit in server mode does not indicate missing JDK
JCE Unlimited Strength Support
Key: NIFI-6254
URL: https://issues.apache.org/jira/browse/NIFI-6254
Project: Apache NiFi
Issue Type: Bug
Components: Security, Tools and Build
Affects Versions: 1.9.2
Environment: JDK 1.8.112
Centos 7
Reporter: Yolanda M. Davis
Assignee: Andy LoPresto
When attempting to run the TLS Toolkit in server mode using a JDK that is not
configured with the JCE Unlimited Strength policy files, the toolkit returns an
error that does not clearly indicate the issue.
For example when running the following command that uses a Java Home without:
${NIFI_TOOLKIT_DIST}/bin/tls-toolkit.sh server -F -f
${NIFI_TOOLKIT_CA_CONFIG_FILE}
The scripts responds with the following:
"Service server error: null"
To surface this error it was necessary to do remote debugging (to see
InvocationException errors were being thrown yet not shown). The request is to
surface this type of error in order for users to take corrective action (in
this case ensuring the JCE policy files are added to the Java SDK Home).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5354) Use ranger-client 1.0.0 in Ranger Authorizer
[ https://issues.apache.org/jira/browse/NIFI-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-5354: --- Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Use ranger-client 1.0.0 in Ranger Authorizer > > > Key: NIFI-5354 > URL: https://issues.apache.org/jira/browse/NIFI-5354 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 1.8.0 > > > We currently use ranger-client 0.7.1, but 1.0.0 has been out for a while. We > should bump the version, and since we also include hadoop-client in the > ranger-authorizer NAR, we should make it so you can override that version > independent of other hadoop.version properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5354) Use ranger-client 1.0.0 in Ranger Authorizer
[ https://issues.apache.org/jira/browse/NIFI-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-5354: --- Affects Version/s: 1.7.0 > Use ranger-client 1.0.0 in Ranger Authorizer > > > Key: NIFI-5354 > URL: https://issues.apache.org/jira/browse/NIFI-5354 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We currently use ranger-client 0.7.1, but 1.0.0 has been out for a while. We > should bump the version, and since we also include hadoop-client in the > ranger-authorizer NAR, we should make it so you can override that version > independent of other hadoop.version properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (NIFI-5354) Use ranger-client 1.0.0 in Ranger Authorizer
[ https://issues.apache.org/jira/browse/NIFI-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-5354: --- Comment: was deleted (was: PR posted here https://github.com/apache/nifi/pull/2827) > Use ranger-client 1.0.0 in Ranger Authorizer > > > Key: NIFI-5354 > URL: https://issues.apache.org/jira/browse/NIFI-5354 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We currently use ranger-client 0.7.1, but 1.0.0 has been out for a while. We > should bump the version, and since we also include hadoop-client in the > ranger-authorizer NAR, we should make it so you can override that version > independent of other hadoop.version properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5354) Use ranger-client 1.0.0 in Ranger Authorizer
[ https://issues.apache.org/jira/browse/NIFI-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-5354: --- Status: Patch Available (was: Open) PR posted here https://github.com/apache/nifi/pull/2827 > Use ranger-client 1.0.0 in Ranger Authorizer > > > Key: NIFI-5354 > URL: https://issues.apache.org/jira/browse/NIFI-5354 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We currently use ranger-client 0.7.1, but 1.0.0 has been out for a while. We > should bump the version, and since we also include hadoop-client in the > ranger-authorizer NAR, we should make it so you can override that version > independent of other hadoop.version properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4942) NiFi Toolkit - Allow migration of master key without previous password
Yolanda M. Davis created NIFI-4942: -- Summary: NiFi Toolkit - Allow migration of master key without previous password Key: NIFI-4942 URL: https://issues.apache.org/jira/browse/NIFI-4942 Project: Apache NiFi Issue Type: Improvement Components: Tools and Build Affects Versions: 1.5.0 Reporter: Yolanda M. Davis Assignee: Andy LoPresto Currently the encryption cli in nifi toolkit requires that, in order to migrate from one master key to the next, the previous master key or password should be provided. In cases where the provisioning tool doesn't have the previous value available this becomes challenging to provide and may be prone to error. In speaking with [~alopresto] we can allow toolkit to support a mode of execution such that the master key can be updated without requiring the previous password. Also documentation around it's usage should be updated to be clear in describing the purpose and the type of environment where this command should be used (admin only access etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4310) Certain flow configurations are improperly identified as empty during flow election
[ https://issues.apache.org/jira/browse/NIFI-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yolanda M. Davis updated NIFI-4310: --- Status: Patch Available (was: In Progress) PR available here: https://github.com/apache/nifi/pull/2107 > Certain flow configurations are improperly identified as empty during flow > election > --- > > Key: NIFI-4310 > URL: https://issues.apache.org/jira/browse/NIFI-4310 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis > > While testing a cluster containing nodes with two different flows on initial > startup, I discovered that one flow, with an empty root processor group but > containing reporting tasks, would lose a flow election against flows that > also had empty root processor groups yet with no reporting tasks or > controller tasks. This caused an issue downstream during startup where it > attempted to replace the non-empty flow with an empty flow on certain nodes, > which led to UninheritableFlowExceptions. Further investigation revealed that > during election there is a check to determine if the flow is empty however > the check does not account for the existence of a reporting task or > controller services. This should be repaired to ensure that flows are > properly identified as non-empty for election. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4310) Certain flow configurations are improperly identified as empty during flow election
Yolanda M. Davis created NIFI-4310: -- Summary: Certain flow configurations are improperly identified as empty during flow election Key: NIFI-4310 URL: https://issues.apache.org/jira/browse/NIFI-4310 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.3.0 Reporter: Yolanda M. Davis Assignee: Yolanda M. Davis While testing a cluster containing nodes with two different flows on initial startup, I discovered that one flow, with an empty root processor group but containing reporting tasks, would lose a flow election against flows that also had empty root processor groups yet with no reporting tasks or controller tasks. This caused an issue downstream during startup where it attempted to replace the non-empty flow with an empty flow on certain nodes, which led to UninheritableFlowExceptions. Further investigation revealed that during election there is a check to determine if the flow is empty however the check does not account for the existence of a reporting task or controller services. This should be repaired to ensure that flows are properly identified as non-empty for election. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
