Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Thanks Mark for the update, I will use Dynatrace to analyse the heap memory and also look for the flows as you suggested. On Fri, Aug 14, 2020 at 7:40 PM Mark Payne wrote: > Sanjeet, > > It’s hard to say what is triggering the OutOfMemoryError. I was able to > create a similarly sized flow and startup just fine using a 2 GB heap. > Since you’ve got 8 GB of heap, it’s unlikely related to the size of just > the flow itself. > > The only way to really definitively diagnose an OutOfMemoryError would be > for you to get a heap dump and analyze that to understand what’s using up > the heap space. > But that said, quite often we the reason that we see OutOfMemoryError is > because users tend to extract FlowFile content into attributes, using > something like ExtractText or EvaluateJsonPath. These processors are > extremely useful for pulling out small pieces of data such as a timestamp > or an “id” field or something like that from data and promoting it to an > attribute. But abusing these types of processors result in huge amounts of > information being added to FlowFile attributes. This then takes up a huge > amount of heap. So if you’re using a lot of that type of pattern, I’d > recommend fixing the flow to avoid that. > > As for an upgrade from 1.8 to 1.11, I would guess that you were already at > the tipping point on 1.8 and just operating below the point of hitting > OutOfMemoryError. They may well be things in 1.11.4 that take a bit more > memory but nothing that I know of that would result in very significant > differences in memory footprint. > > Thanks > -Mark > > On Aug 14, 2020, at 9:18 AM, sanjeet rath wrote: > > Hi Mark/Team > > Any thoughts on this? Where should i analyse further. > > > Regards, > Sanjeet > > > On Thu, 13 Aug 2020, 10:32 pm sanjeet rath, > wrote: > >> Hi Mark, >> Thanks for the response. >> >> My flow.xml.gz file size is 14 Mb. >> >> Regards, >> Sanjeet >> >> On Thu, 13 Aug 2020, 10:27 pm Mark Payne, wrote: >> >>> Actually, I take back what I said. I was a little too quick to jump to >>> conclusions about what the issue was. There was an issue addressed that >>> should improve startup time. But what you’re seeing here is unrelated, as >>> you’re enchanting OutOfMemoryError. The long time is likely related to >>> garbage collection. How large is your flow.xml.gz file? >>> >>> Thanks >>> -Mark >>> >>> >>> On Aug 13, 2020, at 12:48 PM, Mark Payne wrote: >>> >>> Sanjeet, >>> >>> I believe this should be addressed in 1.12.0, which should be released >>> very soon. >>> >>> Thanks >>> -Mark >>> >>> On Aug 13, 2020, at 4:13 AM, sanjeet rath >>> wrote: >>> >>> Hi Team, >>> >>> I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 >>> env to 1.11.4 environment .(12k processor are therr in flow) >>> There are no error in the log file, the issue is taking 10 mintues to >>> server up and with all node connected. >>> >>> The warning logs , which i am suspecting is causing the delay are >>> mentioned below >>> -> I have set max & min jvm to 8gb >>> ->nifi.cluster.node.connection.timeout=30 sec >>> >nifi.cluster.node.read timeout=30 sec >>> ->nifi-zookeeper.connect.timeout=15 sec >>> ->nifi-zookeeper.session.timeout=15 sec >>> >>> When i am making the defaut value 5 & 3 sec respectively, it taking >>> much longer time to node connected. >>> >>> Could you please help me to identify the issue, why it taking so long 10 >>> minutes time to up the nifi cluster.Thanks In advance. >>> >>> 1st suspect warning:(appearing 6 to 7 times in log and dissapears one >>> server is up with nodes are connected) >>> >>> WARN [Process Cluster Protocol Request-10] >>> o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message >>> from “**HOSTIP address**”com due to >>> org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling >>> protocol message in response to message type: CONNECTION_REQUEST due to >>> javax.net.ssl.SSLException: Broken pipe (Write failed) >>> org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling >>> protocol message in response to message type: CONNECTION_REQUEST due to >>> javax.net.ssl.SSLException: Broken pipe (Write failed) >>> at >>> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) >>> at >>> org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:834) >>> Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) >>> at >>> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) >>> at >>> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) >>> at >>> java.base/sun.security.ssl.T
Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Sanjeet, It’s hard to say what is triggering the OutOfMemoryError. I was able to create a similarly sized flow and startup just fine using a 2 GB heap. Since you’ve got 8 GB of heap, it’s unlikely related to the size of just the flow itself. The only way to really definitively diagnose an OutOfMemoryError would be for you to get a heap dump and analyze that to understand what’s using up the heap space. But that said, quite often we the reason that we see OutOfMemoryError is because users tend to extract FlowFile content into attributes, using something like ExtractText or EvaluateJsonPath. These processors are extremely useful for pulling out small pieces of data such as a timestamp or an “id” field or something like that from data and promoting it to an attribute. But abusing these types of processors result in huge amounts of information being added to FlowFile attributes. This then takes up a huge amount of heap. So if you’re using a lot of that type of pattern, I’d recommend fixing the flow to avoid that. As for an upgrade from 1.8 to 1.11, I would guess that you were already at the tipping point on 1.8 and just operating below the point of hitting OutOfMemoryError. They may well be things in 1.11.4 that take a bit more memory but nothing that I know of that would result in very significant differences in memory footprint. Thanks -Mark On Aug 14, 2020, at 9:18 AM, sanjeet rath mailto:rath.sanj...@gmail.com>> wrote: Hi Mark/Team Any thoughts on this? Where should i analyse further. Regards, Sanjeet On Thu, 13 Aug 2020, 10:32 pm sanjeet rath, mailto:rath.sanj...@gmail.com>> wrote: Hi Mark, Thanks for the response. My flow.xml.gz file size is 14 Mb. Regards, Sanjeet On Thu, 13 Aug 2020, 10:27 pm Mark Payne, mailto:marka...@hotmail.com>> wrote: Actually, I take back what I said. I was a little too quick to jump to conclusions about what the issue was. There was an issue addressed that should improve startup time. But what you’re seeing here is unrelated, as you’re enchanting OutOfMemoryError. The long time is likely related to garbage collection. How large is your flow.xml.gz file? Thanks -Mark On Aug 13, 2020, at 12:48 PM, Mark Payne mailto:marka...@hotmail.com>> wrote: Sanjeet, I believe this should be addressed in 1.12.0, which should be released very soon. Thanks -Mark On Aug 13, 2020, at 4:13 AM, sanjeet rath mailto:rath.sanj...@gmail.com>> wrote: Hi Team, I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env to 1.11.4 environment .(12k processor are therr in flow) There are no error in the log file, the issue is taking 10 mintues to server up and with all node connected. The warning logs , which i am suspecting is causing the delay are mentioned below -> I have set max & min jvm to 8gb ->nifi.cluster.node.connection.timeout=30 sec >nifi.cluster.node.read timeout=30 sec ->nifi-zookeeper.connect.timeout=15 sec ->nifi-zookeeper.session.timeout=15 sec When i am making the defaut value 5 & 3 sec respectively, it taking much longer time to node connected. Could you please help me to identify the issue, why it taking so long 10 minutes time to up the nifi cluster.Thanks In advance. 1st suspect warning:(appearing 6 to 7 times in log and dissapears one server is up with nodes are connected) WARN [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from “**HOSTIP address**”com due to org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980) at java.base/java.io.DataOutputStream.write(DataOutputStream.java:107) at java.base/java.io.FilterOutputStream.write
Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Hi Mark/Team Any thoughts on this? Where should i analyse further. Regards, Sanjeet On Thu, 13 Aug 2020, 10:32 pm sanjeet rath, wrote: > Hi Mark, > Thanks for the response. > > My flow.xml.gz file size is 14 Mb. > > Regards, > Sanjeet > > On Thu, 13 Aug 2020, 10:27 pm Mark Payne, wrote: > >> Actually, I take back what I said. I was a little too quick to jump to >> conclusions about what the issue was. There was an issue addressed that >> should improve startup time. But what you’re seeing here is unrelated, as >> you’re enchanting OutOfMemoryError. The long time is likely related to >> garbage collection. How large is your flow.xml.gz file? >> >> Thanks >> -Mark >> >> >> On Aug 13, 2020, at 12:48 PM, Mark Payne wrote: >> >> Sanjeet, >> >> I believe this should be addressed in 1.12.0, which should be released >> very soon. >> >> Thanks >> -Mark >> >> On Aug 13, 2020, at 4:13 AM, sanjeet rath wrote: >> >> Hi Team, >> >> I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 >> env to 1.11.4 environment .(12k processor are therr in flow) >> There are no error in the log file, the issue is taking 10 mintues to >> server up and with all node connected. >> >> The warning logs , which i am suspecting is causing the delay are >> mentioned below >> -> I have set max & min jvm to 8gb >> ->nifi.cluster.node.connection.timeout=30 sec >> >nifi.cluster.node.read timeout=30 sec >> ->nifi-zookeeper.connect.timeout=15 sec >> ->nifi-zookeeper.session.timeout=15 sec >> >> When i am making the defaut value 5 & 3 sec respectively, it taking >> much longer time to node connected. >> >> Could you please help me to identify the issue, why it taking so long 10 >> minutes time to up the nifi cluster.Thanks In advance. >> >> 1st suspect warning:(appearing 6 to 7 times in log and dissapears one >> server is up with nodes are connected) >> >> WARN [Process Cluster Protocol Request-10] >> o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message >> from “**HOSTIP address**”com due to >> org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling >> protocol message in response to message type: CONNECTION_REQUEST due to >> javax.net.ssl.SSLException: Broken pipe (Write failed) >> org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling >> protocol message in response to message type: CONNECTION_REQUEST due to >> javax.net.ssl.SSLException: Broken pipe (Write failed) >> at >> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) >> at >> org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> at java.base/java.lang.Thread.run(Thread.java:834) >> Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) >> at >> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) >> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) >> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) >> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) >> at >> java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980) >> at >> java.base/java.io.DataOutputStream.write(DataOutputStream.java:107) >> at >> java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108) >> at >> org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:86) >> at >> org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182) >> ... 4 common frames omitted >> Suppressed: java.net.SocketException: Broken pipe (Write failed) >> at java.base/java.net.SocketOutputStream.socketWrite0(Native >> Method) >> at >> java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) >> at >> java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) >> at >> java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) >> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357) >> ... 11 common frames omitted >> Caused by: java.net.SocketException: Broken pipe (Write failed) >> at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) >> at >> java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) >> at >> java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) >> at >> java.base/sun.security.ssl.SSLSocketOutputRecord.del
Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Hi Mark, Thanks for the response. My flow.xml.gz file size is 14 Mb. Regards, Sanjeet On Thu, 13 Aug 2020, 10:27 pm Mark Payne, wrote: > Actually, I take back what I said. I was a little too quick to jump to > conclusions about what the issue was. There was an issue addressed that > should improve startup time. But what you’re seeing here is unrelated, as > you’re enchanting OutOfMemoryError. The long time is likely related to > garbage collection. How large is your flow.xml.gz file? > > Thanks > -Mark > > > On Aug 13, 2020, at 12:48 PM, Mark Payne wrote: > > Sanjeet, > > I believe this should be addressed in 1.12.0, which should be released > very soon. > > Thanks > -Mark > > On Aug 13, 2020, at 4:13 AM, sanjeet rath wrote: > > Hi Team, > > I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env > to 1.11.4 environment .(12k processor are therr in flow) > There are no error in the log file, the issue is taking 10 mintues to > server up and with all node connected. > > The warning logs , which i am suspecting is causing the delay are > mentioned below > -> I have set max & min jvm to 8gb > ->nifi.cluster.node.connection.timeout=30 sec > >nifi.cluster.node.read timeout=30 sec > ->nifi-zookeeper.connect.timeout=15 sec > ->nifi-zookeeper.session.timeout=15 sec > > When i am making the defaut value 5 & 3 sec respectively, it taking > much longer time to node connected. > > Could you please help me to identify the issue, why it taking so long 10 > minutes time to up the nifi cluster.Thanks In advance. > > 1st suspect warning:(appearing 6 to 7 times in log and dissapears one > server is up with nodes are connected) > > WARN [Process Cluster Protocol Request-10] > o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from > “**HOSTIP address**”com due to > org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling > protocol message in response to message type: CONNECTION_REQUEST due to > javax.net.ssl.SSLException: Broken pipe (Write failed) > org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling > protocol message in response to message type: CONNECTION_REQUEST due to > javax.net.ssl.SSLException: Broken pipe (Write failed) > at > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) > at > org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) > at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) > at > java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980) > at java.base/java.io.DataOutputStream.write(DataOutputStream.java:107) > at > java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108) > at > org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:86) > at > org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182) > ... 4 common frames omitted > Suppressed: java.net.SocketException: Broken pipe (Write failed) > at java.base/java.net.SocketOutputStream.socketWrite0(Native > Method) > at > java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) > at > java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) > at > java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357) > ... 11 common frames omitted > Caused by: java.net.SocketException: Broken pipe (Write failed) > at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) > at > java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) > at > java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319) > at > java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:975) > ... 8 common frames omitted > 2020-08-13 00:23:51,002 WARN [Process Cluster Protocol Request-6] > org.apache.nifi.io.
Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Actually, I take back what I said. I was a little too quick to jump to conclusions about what the issue was. There was an issue addressed that should improve startup time. But what you’re seeing here is unrelated, as you’re enchanting OutOfMemoryError. The long time is likely related to garbage collection. How large is your flow.xml.gz file? Thanks -Mark On Aug 13, 2020, at 12:48 PM, Mark Payne mailto:marka...@hotmail.com>> wrote: Sanjeet, I believe this should be addressed in 1.12.0, which should be released very soon. Thanks -Mark On Aug 13, 2020, at 4:13 AM, sanjeet rath mailto:rath.sanj...@gmail.com>> wrote: Hi Team, I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env to 1.11.4 environment .(12k processor are therr in flow) There are no error in the log file, the issue is taking 10 mintues to server up and with all node connected. The warning logs , which i am suspecting is causing the delay are mentioned below -> I have set max & min jvm to 8gb ->nifi.cluster.node.connection.timeout=30 sec >nifi.cluster.node.read timeout=30 sec ->nifi-zookeeper.connect.timeout=15 sec ->nifi-zookeeper.session.timeout=15 sec When i am making the defaut value 5 & 3 sec respectively, it taking much longer time to node connected. Could you please help me to identify the issue, why it taking so long 10 minutes time to up the nifi cluster.Thanks In advance. 1st suspect warning:(appearing 6 to 7 times in log and dissapears one server is up with nodes are connected) WARN [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from “**HOSTIP address**”com due to org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980) at java.base/java.io.DataOutputStream.write(DataOutputStream.java:107) at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108) at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:86) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182) ... 4 common frames omitted Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ... 11 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:975) ... 8 common frames omitted 2020-08-13 00:23:51,002 WARN [Process Cluster Protocol Request-6] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2nd suspecting: WARN [Process Cluster Protocol Request-6] org.apache.nifi.io.socket.SocketListen
Re: Nifi 1.11.4 three node cluster is taking longer time to up after migration from 1.8
Sanjeet, I believe this should be addressed in 1.12.0, which should be released very soon. Thanks -Mark On Aug 13, 2020, at 4:13 AM, sanjeet rath mailto:rath.sanj...@gmail.com>> wrote: Hi Team, I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env to 1.11.4 environment .(12k processor are therr in flow) There are no error in the log file, the issue is taking 10 mintues to server up and with all node connected. The warning logs , which i am suspecting is causing the delay are mentioned below -> I have set max & min jvm to 8gb ->nifi.cluster.node.connection.timeout=30 sec >nifi.cluster.node.read timeout=30 sec ->nifi-zookeeper.connect.timeout=15 sec ->nifi-zookeeper.session.timeout=15 sec When i am making the defaut value 5 & 3 sec respectively, it taking much longer time to node connected. Could you please help me to identify the issue, why it taking so long 10 minutes time to up the nifi cluster.Thanks In advance. 1st suspect warning:(appearing 6 to 7 times in log and dissapears one server is up with nodes are connected) WARN [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from “**HOSTIP address**”com due to org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to javax.net.ssl.SSLException: Broken pipe (Write failed) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184) at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980) at java.base/java.io.DataOutputStream.write(DataOutputStream.java:107) at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108) at org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:86) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182) ... 4 common frames omitted Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ... 11 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319) at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:975) ... 8 common frames omitted 2020-08-13 00:23:51,002 WARN [Process Cluster Protocol Request-6] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2nd suspecting: WARN [Process Cluster Protocol Request-6] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3745) at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120) at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputS
