Hi Joe Witt, Yes, first we want to identify actual underlying problem before splitting up NIFI cluster.
Recently observed below scenario as well in the same cluster? We have too many small flow files all the time, on average 5M-10M flow files. NIFI UI/REST API response became too slow & one node got disconnected from cluster & this node is hosting additional DistributedMapCacheServer functionality when compared with other nodes. According to log, it’s disconnected with reason “local flow different that cluster flow” by giving fingerprint information. It looks strange that how a node can disconnect while in running state with this reason, heartbeats are healthy even just before 1 minute of disconnection !! Is there any chance for flow file repository check point interval triggering this issue & automated flow validation at regular intervals with coordinator ? Summary of problem statement as below. 1. NIFI taking huge time to restart (~approx 36 hours) 2. Slower response from NIFI UI/REST API after coming up 3. And then node got disconnected due to local flow different than cluster flow. NIFI app logs from disconnected node as below. Heartbeat Logs : [nifi@HOSTNAME nifi]$ grep -ia "Heartbeat created at" nifi-app_2021-06-29_17* nifi-app_2021-06-29_17.0.log:2021-06-29 17:01:20,923 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 17:01:20,788 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:01:20,923; send took 134 millis nifi-app_2021-06-29_17.0.log:2021-06-29 17:04:21,088 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 17:04:20,967 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:04:21,088; send took 120 millis nifi-app_2021-06-29_17.0.log:2021-06-29 17:07:21,248 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 17:07:21,122 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:07:21,248; send took 125 millis nifi-app_2021-06-29_17.0.log:2021-06-29 17:10:21,461 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 17:10:21,297 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:10:21,461; send took 163 millis nifi-app_2021-06-29_17.0.log:2021-06-29 17:14:13,941 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 17:14:12,343 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:14:13,941; send took 1597 millis Logs showing disconnect message : 2021-06-29 17:15:08,891 INFO [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to properly handle Reconnection request due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. 2021-06-29 17:15:08,957 DEBUG [List FlowFiles for Connection 0826d554-0179-1000-ffff-ffffb244a002] o.a.n.c.queue.AbstractFlowFileQueue org.apache.nifi.controller.queue.AbstractFlowFileQueue$1@45a7cc59<mailto:org.apache.nifi.controller.queue.AbstractFlowFileQueue$1@45a7cc59> Finished listing FlowFiles for active queue with a total of 0 results out of 0 FlowFiles 2021-06-29 17:15:08,961 ERROR [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Handling reconnection request failed due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026) at org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:668) at org.apache.nifi.controller.StandardFlowService.access$200(StandardFlowService.java:109) at org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:415) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows: Local Fingerprint: _REQ_HIST::Transformation::RecordsCount::')}::FileName::${filename}::serviceName::${serviceName}::57bcc7ce-017a-1000-0000-000023992c43NO_VALUETZ_CBS_DB_SRC_CB_GIFT_BOLTON_REQ_HIST::InputFormat for TZ_ Cluster Fingerprint: _REQ_HIST::Transformation::RecordsCount::')}::FileName::${filename}::serviceName::${serviceName}::57caaf2d-017a-1000-0000-000002395ac7NO_VALUEorg.apache.nifi.processors.standard.LogAttributeNO_VALUEor at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:293) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1001) ... 4 common frames omitted 2021-06-29 17:15:08,961 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator HOSTNAME:9091 requested disconnection from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. Thanks & Regards, Avinash M V From: Joe Witt <[email protected]> Sent: 30 June 2021 07:58 To: [email protected] Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster CAUTION: This e-mail originated from outside of the organization. Do not click links or open attachments unless you recognise the sender and know the content is safe. I dont recommend splitting the clusters for this. Lets figure out what is happening for your case. On Tue, Jun 29, 2021 at 10:25 PM Modepalli Venkata Avinash <[email protected]<mailto:[email protected]>> wrote: Hi Joe, Thanks for your feedback. Yes, working on to split into multiple clusters. But it looks like this problem will be still there while restarting NIFI cluster if flow.xml.gz size is grows in 10’s of MBs (When more number of components/processors deployed on root canvas increases). Thanks & Regards, Avinash M V From: Joe Obernberger <[email protected]<mailto:[email protected]>> Sent: 29 June 2021 17:09 To: [email protected]<mailto:[email protected]>; Modepalli Venkata Avinash <[email protected]<mailto:[email protected]>> Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster CAUTION: This e-mail originated from outside of the organization. Do not click links or open attachments unless you recognise the sender and know the content is safe. Impressive cluster size! I do not have an answer for you, but could you change your architecture so that instead of one large NiFi cluster you have 2 or 3 smaller clusters? Very curious on the answer here as I have also noticed UI slow-downs as the number of nodes increases. -Joe On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote: Hi List, We have 13 Nodes NIFI cluster in production & it’s taking huge time for completing NIFI restart. According to our analysis, flow election & flow validation from other nodes with coordinator is taking more time, approx. ~30 hours. Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please find below cluster details. Apache NIFI Version : 1.9.0 Flow.xml.gz size : 13MB (gz compressed) OS : RHEL 7.6 JDK : jdk1.8.0_151 GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC because of Numerous bugs in JDK8 while using with WriteaHeadProvenance Repository Min & Max Memory : 140GB Server Memory Per Node : 256GB CPU/CORE : 48 Number of Nodes in Cluster : 13 Max Timer Driven Thread : 100 Running Processors Count : 12K Stopped Processors Count : 10K Disabled Processors Count : 25K Total Processor Count : 47K We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. Could you please share any input to identify & resolve this issue. Thanks for your help. Thanks & Regards, Avinash M V [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
