[ https://issues.apache.org/jira/browse/NIFI-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yolanda M. Davis updated NIFI-7437: ----------------------------------- Status: Patch Available (was: Open) > UI is slow when nifi.analytics.predict.enabled is true > ------------------------------------------------------ > > Key: NIFI-7437 > URL: https://issues.apache.org/jira/browse/NIFI-7437 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI, Extensions > Affects Versions: 1.11.4, 1.10.0 > Environment: Java11, CentOS8 > Reporter: Dmitry Ibragimov > Assignee: Yolanda M. Davis > Priority: Critical > Labels: features, performance > Time Spent: 10m > Remaining Estimate: 0h > > We faced with issue when nifi.analytics.predict.enabled is true after cluster > upgrade to 1.11.4 > We have about 4000 processors in development enviroment, but most of them is > in disabled state: 256 running, 1263 stopped, 2543 disabled > After upgrade version from 1.9.2 to 1.11.4 we deicded to test back-pressure > prediction feature and enable it in configuration: > {code:java} > nifi.analytics.predict.enabled=true > nifi.analytics.predict.interval=3 mins > nifi.analytics.query.interval=5 mins > nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares > nifi.analytics.connection.model.score.name=rSquared > nifi.analytics.connection.model.score.threshold=.90 > {code} > And we faced with terrible UI performance degradataion. Root page opens in 20 > seconds instead of 200-500ms. About ~100 times slower. I've tesed it with > different environments centos7/8, java8/11, clustered secured, clustered > unsecured, standalone unsecured - all the same. > In debug log for ThreadPoolRequestReplicator: > {code:java} > 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator For GET > /nifi-api/flow/process-groups/root (Request ID > c144196f-d4cb-4053-8828-70e06f7c5100), minimum response time = 19548, max = > 20625, average = 20161.0 ms > 2020-05-09 08:03:34,459 DEBUG [Replicate Request Thread-2] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET > /nifi-api/flow/process-groups/root (Request ID > c144196f-d4cb-4053-8828-70e06f7c5100): > newnifi01:8080: 19548 millis > newnifi02:8080: 20625 millis > newnifi03:8080: 20310 millis{code} > More deep debug: > > {code:java} > 2020-05-09 10:31:13,252 DEBUG [NiFi Web Server-21] > org.eclipse.jetty.server.HttpChannel REQUEST for > //newnifi01:8080/nifi-api/flow/process-groups/root on > HttpChannelOverHttp@68d3e945{r=1,c=false,c=false/false,a=IDLE,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=0} > GET //newnifi01:8080/nifi-api/flow/process-groups/root HTTP/1.1 > Host: newnifi01:8080 > ... > 2020-05-09 10:31:13,256 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time > back pressure by content size in bytes. Returning -1 > 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time > to back pressure by object count. Returning -1 > 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > content size in bytes for next interval. Returning -1 > 2020-05-09 10:31:13,257 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > object count for next interval. Returning -1 > 2020-05-09 10:31:13,258 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > object count for next interval. Returning -1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > content size in bytes for next interval. Returning -1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: nextIntervalPercentageUseCount=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: nextIntervalBytes=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: timeToBytesBackpressureMillis=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: nextIntervalCount=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: nextIntervalPercentageUseBytes=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: intervalTimeMillis=180000 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > eb602b2a-016f-1000-0000-00002767192a: timeToCountBackpressureMillis=-1 > 2020-05-09 10:31:13,259 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.CachingConnectionStatusAnalyticsEngine Pulled existing analytics > from cache for connection id: ec014ca8-a82b-10bb-0000-00004069f95e > 2020-05-09 10:31:13,260 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time > back pressure by content size in bytes. Returning -1 > 2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time > to back pressure by object count. Returning -1 > 2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > content size in bytes for next interval. Returning -1 > 2020-05-09 10:31:13,261 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > object count for next interval. Returning -1 > 2020-05-09 10:31:13,262 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > object count for next interval. Returning -1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for predicting > content size in bytes for next interval. Returning -1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalPercentageUseCount=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalBytes=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: timeToBytesBackpressureMillis=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalCount=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: nextIntervalPercentageUseBytes=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: intervalTimeMillis=180000 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Prediction model for connection id > ec014ca8-a82b-10bb-0000-00004069f95e: timeToCountBackpressureMillis=-1 > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.CachingConnectionStatusAnalyticsEngine Pulled existing analytics > from cache for connection id: eb61f002-016f-1000-0000-000067a860ea > 2020-05-09 10:31:13,263 DEBUG [NiFi Web Server-21] > o.a.n.c.s.a.ConnectionStatusAnalytics Model is not valid for calculating time > back pressure by content size in bytes. Returning -1 > ... > A lot of same messages > ... > 2020-05-09 10:31:32,758 DEBUG [NiFi Web Server-21] > org.eclipse.jetty.server.HttpChannel COMMIT for > /nifi-api/flow/process-groups/root on > HttpChannelOverHttp@68d3e945{r=1,c=true,c=false/false,a=DISPATCHED,uri=//newnifi01:8080/nifi-api/flow/process-groups/root,age=19506} > 200 OK HTTP/1.1 > Date: Sat, 09 May 2020 07:31:13 GMT > {code} > 3ms prediction is a quite fast, but if calculate count this messages it is > the same as number of processors: > {code:java} > #zgrep "2020-05-09 10:31:" /var/log/nifi/nifi-app_2020-05-09_10.1.log.gz | > grep "Prediction model for connection id" | cut -d' ' -f 13 | sort | uniq -c > | wc -l > 4842{code} > I've checked several random connection id's - it is from and to disabled > processors. > Maybe need to skip prediction back-pressure for disbled connections? Any > suppouse, how can we fix it without disable prediction entirely? > -- This message was sent by Atlassian Jira (v8.3.4#803005)