RE: Taking Huge Time for Connect all Nodes to NIFI cluster

[Modepalli Venkata Avinash]

Hi Vijay,

Thanks for your inputs.

Actually we found that there is approximation formula for this setting. 
Recommended configuration for this 2-4 times than number of cores on machine.
We have 48 cores on machine (multithreading enabled), hence configured 100 Max 
Timer Driven Threads at this point of time.
According to our understanding, this configuration is applied per node. As we 
configured 100 Max Timer Driven Threads, so our cluster can consume maximum 
13*100=1300 Max Timer Driven Threads (Number of Nodes * Max Timer Driven 
Threads)
Reference Link shared by you also conveys that Max Timer Driven Threads can 
increase to 72 based upon need when 24 is configured in NIFI.

https://community.cloudera.com/t5/Support-Questions/How-to-configure-NiFi-to-maximize-the-usage-of-system-cores/td-p/189581
https://marklogic.github.io/nifi/performance-considerations

Thanks & Regards,
Avinash M V

From: Vijay Chhipa 
Sent: 30 June 2021 08:11
To: users@nifi.apache.org
Cc: Joe Witt 
Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster

Avinash,

Did you try to tune Max Timer Driven Thread : 100

Set this value to  (number of cores per node) * (number of nodes) i.e. 48 *13 = 
624

https://docs.cloudera.com/cfm/2.1.1/nifi-tuning/topics/cfm-tuning-recommendations.html


HTH




On Jun 29, 2021, at 9:32 PM, Modepalli Venkata Avinash 
mailto:modepalli.avin...@subex.com>> wrote:

Hi Joe,

Thanks for your feedback.

It’s deployed by using Hortonworks stack, so if we have to upgrade our NIFI 
then we might have to upgrade entire stack. Also we have some integrations done 
to NIFI using REST API along with some custom processors.
We have plans to upgrade down the lane, but not immediately due to above 
challenges.
We have almost 5-10M flow files in queues while bringing up NIFI. However, we 
are observing this issue even with lesser number of flow files also.

Thanks & Regards,
Avinash M V

From: Joe Witt mailto:joe.w...@gmail.com>>
Sent: 29 June 2021 17:15
To: users@nifi.apache.org
Cc: Modepalli Venkata Avinash 
mailto:modepalli.avin...@subex.com>>
Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster

CAUTION: This e-mail originated from outside of the organization. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

Hello

A cluster that size should be fine. We did make various improvement to cluster 
behavior and startup times though.  What prevents you from moving to 1.13?

How many flowfiles are in the repo when restarting is taking that long?

thanks

On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:
Impressive cluster size!  I do not have an answer for you, but could you change 
your architecture so that instead of one large NiFi cluster you have 2 or 3 
smaller clusters?  Very curious on the answer here as I have also noticed UI 
slow-downs as the number of nodes increases.
-Joe
On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
Hi List,

We have 13 Nodes NIFI cluster in production & it’s taking huge time for 
completing NIFI restart.
According to our analysis, flow election & flow validation from other nodes 
with coordinator is taking more time, approx. ~30 hours.
Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
find below cluster details.

Apache NIFI Version : 1.9.0
Flow.xml.gz size : 13MB (gz compressed)
OS : RHEL 7.6
JDK : jdk1.8.0_151
GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC because 
of Numerous bugs in JDK8 while using with WriteaHeadProvenance Repository
Min & Max Memory : 140GB
Server Memory Per Node : 256GB
CPU/CORE : 48
Number of Nodes in Cluster : 13
Max Timer Driven Thread : 100
Running Processors Count : 12K
Stopped Processors Count : 10K
Disabled Processors Count : 25K
Total Processor Count : 47K

We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. 
Could you please share any input to identify & resolve this issue.
Thanks for your help.

Thanks & Regards,
Avinash M V


[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]
Virus-free. 
www.avg.com

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

[Modepalli Venkata Avinash]

Hi Joe Witt,

Yes, first we want to identify actual underlying problem before splitting up 
NIFI cluster.

Recently observed below scenario as well in the same cluster?

We have too many small flow files all the time, on average 5M-10M flow files. 
NIFI UI/REST API response became too slow & one node got disconnected from 
cluster & this node is hosting additional DistributedMapCacheServer 
functionality when compared with other nodes. According to log, it’s 
disconnected with reason “local flow different that cluster flow” by giving 
fingerprint information.

It looks strange that how a node can disconnect while in running state with 
this reason, heartbeats are healthy even just before 1 minute of disconnection 
!!
Is there any chance for flow file repository check point interval triggering 
this issue & automated flow validation at regular intervals with coordinator ? 
Summary of problem statement as below.


  1.  NIFI taking huge time to restart (~approx 36 hours)
  2.  Slower response from NIFI UI/REST API after coming up
  3.  And then node got disconnected due to local flow different than cluster 
flow.

NIFI app logs from disconnected node as below.

Heartbeat Logs :

[nifi@HOSTNAME nifi]$ grep -ia "Heartbeat created at" nifi-app_2021-06-29_17*
nifi-app_2021-06-29_17.0.log:2021-06-29 17:01:20,923 INFO [Clustering Tasks 
Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 
17:01:20,788 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:01:20,923; 
send took 134 millis
nifi-app_2021-06-29_17.0.log:2021-06-29 17:04:21,088 INFO [Clustering Tasks 
Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 
17:04:20,967 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:04:21,088; 
send took 120 millis
nifi-app_2021-06-29_17.0.log:2021-06-29 17:07:21,248 INFO [Clustering Tasks 
Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 
17:07:21,122 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:07:21,248; 
send took 125 millis
nifi-app_2021-06-29_17.0.log:2021-06-29 17:10:21,461 INFO [Clustering Tasks 
Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 
17:10:21,297 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:10:21,461; 
send took 163 millis
nifi-app_2021-06-29_17.0.log:2021-06-29 17:14:13,941 INFO [Clustering Tasks 
Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2021-06-29 
17:14:12,343 and sent to COORDINATOR_HOSTNAME:9088 at 2021-06-29 17:14:13,941; 
send took 1597 millis

Logs showing disconnect message :

2021-06-29 17:15:08,891 INFO [Reconnect to Cluster] 
o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to 
properly handle Reconnection request due to 
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.

2021-06-29 17:15:08,957 DEBUG [List FlowFiles for Connection 
0826d554-0179-1000--b244a002] o.a.n.c.queue.AbstractFlowFileQueue 
org.apache.nifi.controller.queue.AbstractFlowFileQueue$1@45a7cc59
 Finished listing FlowFiles for active queue with a total of 0 results out of 0 
FlowFiles
2021-06-29 17:15:08,961 ERROR [Reconnect to Cluster] 
o.a.nifi.controller.StandardFlowService Handling reconnection request failed 
due to: org.apache.nifi.controller.UninheritableFlowException: Failed to 
connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026)
at 
org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:668)
at 
org.apache.nifi.controller.StandardFlowService.access$200(StandardFlowService.java:109)
at 
org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:415)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
configuration is not inheritable by the flow controller because of flow 
differences: Found difference in Flows:
Local Fingerprint:   
_REQ_HIST::Transformation::RecordsCount::')}::FileName::${filename}::serviceName::${serviceName}::57bcc7ce-017a-1000--23992c43NO_VALUETZ_CBS_DB_SRC_CB_GIFT_BOLTON_REQ_HIST::InputFormat
 for TZ_
Cluster Fingerprint: 
_REQ_HIST::Transformation::RecordsCount::')}::FileName::${filename}::serviceName::${serviceName}::57caaf2d-017a-1000--02395ac7NO_VALUEorg.apache.nifi.processors.standard.LogAttributeNO_VALUEor
at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:293)
at 

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

[Vijay Chhipa]

Avinash, 

Did you try to tune Max Timer Driven Thread : 100

Set this value to  (number of cores per node) * (number of nodes) i.e. 48 *13 = 
624

https://docs.cloudera.com/cfm/2.1.1/nifi-tuning/topics/cfm-tuning-recommendations.html
 



HTH  



> On Jun 29, 2021, at 9:32 PM, Modepalli Venkata Avinash 
>  wrote:
> 
> Hi Joe,
>  
> Thanks for your feedback.
>  
> It’s deployed by using Hortonworks stack, so if we have to upgrade our NIFI 
> then we might have to upgrade entire stack. Also we have some integrations 
> done to NIFI using REST API along with some custom processors. 
> We have plans to upgrade down the lane, but not immediately due to above 
> challenges.
> We have almost 5-10M flow files in queues while bringing up NIFI. However, we 
> are observing this issue even with lesser number of flow files also.
>  
> Thanks & Regards,
> Avinash M V
>  
> From: Joe Witt mailto:joe.w...@gmail.com>> 
> Sent: 29 June 2021 17:15
> To: users@nifi.apache.org 
> Cc: Modepalli Venkata Avinash  >
> Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster
>  
> CAUTION: This e-mail originated from outside of the organization. Do not 
> click links or open attachments unless you recognise the sender and know the 
> content is safe.
>  
> Hello
>  
> A cluster that size should be fine. We did make various improvement to 
> cluster behavior and startup times though.  What prevents you from moving to 
> 1.13?
>  
> How many flowfiles are in the repo when restarting is taking that long? 
>  
> thanks
>  
> On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger  > wrote:
> Impressive cluster size!  I do not have an answer for you, but could you 
> change your architecture so that instead of one large NiFi cluster you have 2 
> or 3 smaller clusters?  Very curious on the answer here as I have also 
> noticed UI slow-downs as the number of nodes increases.
> 
> -Joe
> 
> On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
> Hi List,
>  
> We have 13 Nodes NIFI cluster in production & it’s taking huge time for 
> completing NIFI restart.
> According to our analysis, flow election & flow validation from other nodes 
> with coordinator is taking more time, approx. ~30 hours.
> Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
> find below cluster details.
>  
> Apache NIFI Version : 1.9.0
> Flow.xml.gz size : 13MB (gz compressed)
> OS : RHEL 7.6
> JDK : jdk1.8.0_151
> GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC 
> because of Numerous bugs in JDK8 while using with WriteaHeadProvenance 
> Repository
> Min & Max Memory : 140GB
> Server Memory Per Node : 256GB
> CPU/CORE : 48
> Number of Nodes in Cluster : 13
> Max Timer Driven Thread : 100
> Running Processors Count : 12K
> Stopped Processors Count : 10K
> Disabled Processors Count : 25K
> Total Processor Count : 47K
>  
> We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. 
> Could you please share any input to identify & resolve this issue.
> Thanks for your help.
>  
> Thanks & Regards,
> Avinash M V
>  
>  
>  
> 
>  
> Virus-free. www.avg.com 
> 


smime.p7s
Description: S/MIME cryptographic signature

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

[Modepalli Venkata Avinash]

Hi Joe,

Thanks for your feedback.

It’s deployed by using Hortonworks stack, so if we have to upgrade our NIFI 
then we might have to upgrade entire stack. Also we have some integrations done 
to NIFI using REST API along with some custom processors.
We have plans to upgrade down the lane, but not immediately due to above 
challenges.
We have almost 5-10M flow files in queues while bringing up NIFI. However, we 
are observing this issue even with lesser number of flow files also.

Thanks & Regards,
Avinash M V

From: Joe Witt 
Sent: 29 June 2021 17:15
To: users@nifi.apache.org
Cc: Modepalli Venkata Avinash 
Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster

CAUTION: This e-mail originated from outside of the organization. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

Hello

A cluster that size should be fine. We did make various improvement to cluster 
behavior and startup times though.  What prevents you from moving to 1.13?

How many flowfiles are in the repo when restarting is taking that long?

thanks

On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Impressive cluster size!  I do not have an answer for you, but could you change 
your architecture so that instead of one large NiFi cluster you have 2 or 3 
smaller clusters?  Very curious on the answer here as I have also noticed UI 
slow-downs as the number of nodes increases.

-Joe
On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
Hi List,

We have 13 Nodes NIFI cluster in production & it’s taking huge time for 
completing NIFI restart.
According to our analysis, flow election & flow validation from other nodes 
with coordinator is taking more time, approx. ~30 hours.
Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
find below cluster details.

Apache NIFI Version : 1.9.0
Flow.xml.gz size : 13MB (gz compressed)
OS : RHEL 7.6
JDK : jdk1.8.0_151
GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC because 
of Numerous bugs in JDK8 while using with WriteaHeadProvenance Repository
Min & Max Memory : 140GB
Server Memory Per Node : 256GB
CPU/CORE : 48
Number of Nodes in Cluster : 13
Max Timer Driven Thread : 100
Running Processors Count : 12K
Stopped Processors Count : 10K
Disabled Processors Count : 25K
Total Processor Count : 47K

We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. 
Could you please share any input to identify & resolve this issue.
Thanks for your help.

Thanks & Regards,
Avinash M V


[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]
Virus-free. 
www.avg.com

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

[Joe Witt]

I dont recommend splitting the clusters for this.  Lets figure out what is
happening for your case.

On Tue, Jun 29, 2021 at 10:25 PM Modepalli Venkata Avinash <
modepalli.avin...@subex.com> wrote:

> Hi Joe,
>
>
>
> Thanks for your feedback.
>
> Yes, working on to split into multiple clusters. But it looks like this
> problem will be still there while restarting NIFI cluster if flow.xml.gz
> size is grows in 10’s of MBs (When more number of components/processors
> deployed on root canvas increases).
>
>
>
> *Thanks & Regards,*
>
> *Avinash M V*
>
>
>
> *From:* Joe Obernberger 
> *Sent:* 29 June 2021 17:09
> *To:* users@nifi.apache.org; Modepalli Venkata Avinash <
> modepalli.avin...@subex.com>
> *Subject:* Re: Taking Huge Time for Connect all Nodes to NIFI cluster
>
>
>
> *CAUTION:* This e-mail originated from outside of the organization. Do
> not click links or open attachments unless you recognise the sender and
> know the content is safe.
>
>
>
> Impressive cluster size!  I do not have an answer for you, but could you
> change your architecture so that instead of one large NiFi cluster you have
> 2 or 3 smaller clusters?  Very curious on the answer here as I have also
> noticed UI slow-downs as the number of nodes increases.
>
> -Joe
>
> On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
>
> Hi List,
>
>
>
> We have 13 Nodes NIFI cluster in production & it’s taking huge time for
> completing NIFI restart.
>
> According to our analysis, flow election & flow validation from other
> nodes with coordinator is taking more time, approx. ~30 hours.
>
> Even after all 13 nodes gets connected, NIFI UI responds too slowly.
> Please find below cluster details.
>
>
>
> Apache NIFI Version : 1.9.0
>
> Flow.xml.gz size : 13MB (gz compressed)
>
> OS : RHEL 7.6
>
> JDK : jdk1.8.0_151
>
> GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC
> because of Numerous bugs in JDK8 while using with WriteaHeadProvenance
> Repository
>
> Min & Max Memory : 140GB
>
> Server Memory Per Node : 256GB
>
> CPU/CORE : 48
>
> Number of Nodes in Cluster : 13
>
> Max Timer Driven Thread : 100
>
> Running Processors Count : 12K
>
> Stopped Processors Count : 10K
>
> Disabled Processors Count : 25K
>
> Total Processor Count : 47K
>
>
>
> We couldn’t find any abnormalities in app logs, bootstrap logs & GC
> logging. Could you please share any input to identify & resolve this issue.
>
> Thanks for your help.
>
>
>
> *Thanks & Regards,*
>
> *Avinash M V*
>
>
>
>
>
>
> 
>
> Virus-free. www.avg.com
> 
>
>
>
>

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

[Modepalli Venkata Avinash]

Hi Joe,

Thanks for your feedback.
Yes, working on to split into multiple clusters. But it looks like this problem 
will be still there while restarting NIFI cluster if flow.xml.gz size is grows 
in 10’s of MBs (When more number of components/processors deployed on root 
canvas increases).

Thanks & Regards,
Avinash M V

From: Joe Obernberger 
Sent: 29 June 2021 17:09
To: users@nifi.apache.org; Modepalli Venkata Avinash 

Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster

CAUTION: This e-mail originated from outside of the organization. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.


Impressive cluster size!  I do not have an answer for you, but could you change 
your architecture so that instead of one large NiFi cluster you have 2 or 3 
smaller clusters?  Very curious on the answer here as I have also noticed UI 
slow-downs as the number of nodes increases.

-Joe
On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
Hi List,

We have 13 Nodes NIFI cluster in production & it’s taking huge time for 
completing NIFI restart.
According to our analysis, flow election & flow validation from other nodes 
with coordinator is taking more time, approx. ~30 hours.
Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
find below cluster details.

Apache NIFI Version : 1.9.0
Flow.xml.gz size : 13MB (gz compressed)
OS : RHEL 7.6
JDK : jdk1.8.0_151
GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC because 
of Numerous bugs in JDK8 while using with WriteaHeadProvenance Repository
Min & Max Memory : 140GB
Server Memory Per Node : 256GB
CPU/CORE : 48
Number of Nodes in Cluster : 13
Max Timer Driven Thread : 100
Running Processors Count : 12K
Stopped Processors Count : 10K
Disabled Processors Count : 25K
Total Processor Count : 47K

We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. 
Could you please share any input to identify & resolve this issue.
Thanks for your help.

Thanks & Regards,
Avinash M V


[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]
Virus-free. 
www.avg.com

Re: NiFi configuration files changes

[Andrew Grande]

The physical files will get synchronized to the reference state from a
central config management source (CM). There's no point watching them on
the file system. If you need a change log for config files, I'd look into
CM api to fetch those instead.

On Tue, Jun 29, 2021, 8:30 AM Tomislav Novosel <
tomislav.novo...@clearpeaks.com> wrote:

> Hi to all,
>
>
>
> Is there a good way how to capture NiFi configuration files changes
> (nifi.properties, authorizers.xml,…etc)
>
> and to forward that changes (or just to notify) some other system or app?
>
> Can I do it with NiFi itself?
>
> The question is in the context of Cloudera platform – CFM.
>
>
>
> Thanks,
>
> Regards,
>
> Tom
>

NiFi configuration files changes

[Tomislav Novosel]

Hi to all,

Is there a good way how to capture NiFi configuration files changes 
(nifi.properties, authorizers.xml,...etc)
and to forward that changes (or just to notify) some other system or app?
Can I do it with NiFi itself?
The question is in the context of Cloudera platform - CFM.

Thanks,
Regards,
Tom

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

[Joe Witt]

Hello

A cluster that size should be fine. We did make various improvement to
cluster behavior and startup times though.  What prevents you from moving
to 1.13?

How many flowfiles are in the repo when restarting is taking that long?

thanks

On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:

> Impressive cluster size!  I do not have an answer for you, but could you
> change your architecture so that instead of one large NiFi cluster you have
> 2 or 3 smaller clusters?  Very curious on the answer here as I have also
> noticed UI slow-downs as the number of nodes increases.
>
> -Joe
> On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
>
> Hi List,
>
>
>
> We have 13 Nodes NIFI cluster in production & it’s taking huge time for
> completing NIFI restart.
>
> According to our analysis, flow election & flow validation from other
> nodes with coordinator is taking more time, approx. ~30 hours.
>
> Even after all 13 nodes gets connected, NIFI UI responds too slowly.
> Please find below cluster details.
>
>
>
> Apache NIFI Version : 1.9.0
>
> Flow.xml.gz size : 13MB (gz compressed)
>
> OS : RHEL 7.6
>
> JDK : jdk1.8.0_151
>
> GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC
> because of Numerous bugs in JDK8 while using with WriteaHeadProvenance
> Repository
>
> Min & Max Memory : 140GB
>
> Server Memory Per Node : 256GB
>
> CPU/CORE : 48
>
> Number of Nodes in Cluster : 13
>
> Max Timer Driven Thread : 100
>
> Running Processors Count : 12K
>
> Stopped Processors Count : 10K
>
> Disabled Processors Count : 25K
>
> Total Processor Count : 47K
>
>
>
> We couldn’t find any abnormalities in app logs, bootstrap logs & GC
> logging. Could you please share any input to identify & resolve this issue.
>
> Thanks for your help.
>
>
>
> *Thanks & Regards,*
>
> *Avinash M V*
>
>
>
>
>
> 
>  Virus-free.
> www.avg.com
> 
> <#m_-1430803108974534026_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

[Joe Obernberger]

Impressive cluster size!  I do not have an answer for you, but could you 
change your architecture so that instead of one large NiFi cluster you 
have 2 or 3 smaller clusters?  Very curious on the answer here as I have 
also noticed UI slow-downs as the number of nodes increases.


-Joe

On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:


Hi List,

We have 13 Nodes NIFI cluster in production & it’s taking huge time 
for completing NIFI restart.


According to our analysis, flow election & flow validation from other 
nodes with coordinator is taking more time, approx. ~30 hours.


Even after all 13 nodes gets connected, NIFI UI responds too slowly. 
Please find below cluster details.


Apache NIFI Version : 1.9.0

Flow.xml.gz size : 13MB (gz compressed)

OS : RHEL 7.6

JDK : jdk1.8.0_151

GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC 
because of Numerous bugs in JDK8 while using with WriteaHeadProvenance 
Repository


Min & Max Memory : 140GB

Server Memory Per Node : 256GB

CPU/CORE : 48

Number of Nodes in Cluster : 13

Max Timer Driven Thread : 100

Running Processors Count : 12K

Stopped Processors Count : 10K

Disabled Processors Count : 25K

Total Processor Count : 47K

We couldn’t find any abnormalities in app logs, bootstrap logs & GC 
logging. Could you please share any input to identify & resolve this 
issue.


Thanks for your help.

*//*

*/Thanks & Regards,/*

*/Avinash M V/*


 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

[Modepalli Venkata Avinash]

Hi List,

We have 13 Nodes NIFI cluster in production & it's taking huge time for 
completing NIFI restart.
According to our analysis, flow election & flow validation from other nodes 
with coordinator is taking more time, approx. ~30 hours.
Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
find below cluster details.

Apache NIFI Version : 1.9.0
Flow.xml.gz size : 13MB (gz compressed)
OS : RHEL 7.6
JDK : jdk1.8.0_151
GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC because 
of Numerous bugs in JDK8 while using with WriteaHeadProvenance Repository
Min & Max Memory : 140GB
Server Memory Per Node : 256GB
CPU/CORE : 48
Number of Nodes in Cluster : 13
Max Timer Driven Thread : 100
Running Processors Count : 12K
Stopped Processors Count : 10K
Disabled Processors Count : 25K
Total Processor Count : 47K

We couldn't find any abnormalities in app logs, bootstrap logs & GC logging. 
Could you please share any input to identify & resolve this issue.
Thanks for your help.

Thanks & Regards,
Avinash M V