Hi All, Happy New Year :)
I have upgraded our cluster from 1.8 to 1.12.1, few days ago and everything is working fine. I observed that Nifi was like hanged after running for few days (I have observed its nearly after 15 days of nifi service start) issue is after login the browser keep on loading , When I saw the bootstrap.log I saw this message "*Apache nifi is running at PID () but not responding to ping requests*”. This happened to only one node from a 3 node cluster. This issue happened *3 times on different cluster on different nodes.* *Everytime issue got fixed by restarting NiFi service.* During the hanged state I tried see the resource utilisation -> top -n 1 -H -p 943785 (nifi processid ) top - 08:26:36 up 40 days, 3:48, 2 users, load average: 5.28, 5.38, 5.43 Threads: 239 total, 4 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): 98.7 us, 1.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 15829.5 total, 610.8 free, 10823.7 used, 4395.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 4456.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND *943806* root 20 0 12.5g 9.4g 18692 R *88.9* 60.7 12698:50 *GC Thread#1 * 943807 root 20 0 12.5g 9.4g 18692 R 88.9 60.7 12698:48 GC Thread#2 943808 root 20 0 12.5g 9.4g 18692 R 88.9 60.7 12698:58 GC Thread#3 943787 root 20 0 12.5g 9.4g 18692 R 83.3 60.7 12698:51 GC Thread#0 943785 root 20 0 12.5g 9.4g 18692 S 0.0 60.7 0:00.00 java We have 4 core cpu, all *4 GC threads* are keep on this state and consuming more CPU.*cluster is hung state for 2 days,* Then after 2 days I saw these threads are moved and nifi comes out of the hung state for this node , but saw another node from the same cluster moved to the hung state with similar fashion means , 4 threads busy in GC and consuming more CPU. Could you please help me to identify what could be the possible reason. Details: Nifi 1.12.1 Jdk 11 Zookeeper 3.5.8 16g memory Thanks, -- Sanjeet Kumar Rath, mob- +91 8777577470
