Hi Team,
I have migrated my flows.xml.gz, users.xml, authorization.xml from 1.8 env
to 1.11.4 environment .(12k processor are therr in flow)
There are no error in the log file, the issue is taking 10 mintues to
server up and with all node connected.
The warning logs , which i am suspecting is causing the delay are mentioned
below
-> I have set max & min jvm to 8gb
->nifi.cluster.node.connection.timeout=30 sec
>nifi.cluster.node.read timeout=30 sec
->nifi-zookeeper.connect.timeout=15 sec
->nifi-zookeeper.session.timeout=15 sec
When i am making the defaut value 5 & 3 sec respectively, it taking
much longer time to node connected.
Could you please help me to identify the issue, why it taking so long 10
minutes time to up the nifi cluster.Thanks In advance.
1st suspect warning:(appearing 6 to 7 times in log and dissapears one
server is up with nodes are connected)
WARN [Process Cluster Protocol Request-10]
o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol
message from “**HOSTIP address**”com due to
org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling
protocol message in response to message type: CONNECTION_REQUEST due
to javax.net.ssl.SSLException: Broken pipe (Write failed)
org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling
protocol message in response to message type: CONNECTION_REQUEST due
to javax.net.ssl.SSLException: Broken pipe (Write failed)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:184)
at
org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at
java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:980)
at java.base/java.io.DataOutputStream.write(DataOutputStream.java:107)
at
java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
at
org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:86)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182)
... 4 common frames omitted
Suppressed: java.net.SocketException: Broken pipe (Write failed)
at
java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at
java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at
java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357)
... 11 common frames omitted
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at
java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at
java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319)
at
java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:975)
... 8 common frames omitted
2020-08-13 00:23:51,002 WARN [Process Cluster Protocol Request-6]
org.apache.nifi.io.socket.SocketListener Dispatching socket request
encountered exception due to: java.lang.OutOfMemoryError: Java heap
space
java.lang.OutOfMemoryError: Java heap space
2nd suspecting:
WARN [Process Cluster Protocol Request-6]
org.apache.nifi.io.socket.SocketListener Dispatching socket request
encountered exception due to: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space at
java.base/java.util.Arrays.copyOf(Arrays.java:3745) at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.flushBuffer(UTF8XmlOutput.java:418)
at
com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.text(UTF8XmlOutput.java:371)
at
com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.writeTo(Base64Data.java:314)
at
com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.text(UTF8XmlOutput.java:312)
at
com.sun.xml.bind.v2.runtime.XMLSerializer.leafElement(XMLSerializer.java:356)
at
com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$PcdataImpl.writeLeafElement(RuntimeBuiltinLeafInfoImpl.java:191)
at
com.sun.xml.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.writeLeafElement(TransducedAccessor.java:254)
at
com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty.serializeBody(SingleElementLeafProperty.java:130)
at
com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:360)
at
com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:696)
at
com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:158)
at
com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:360)
at
com.sun.xml.bind.v2.runtime.XMLSerializer.childAsXsiType(XMLSerializer.java:696)
at
com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.serializeBody(SingleElementNodeProperty.java:158)
at
com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:360)
at
com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:593)
at
com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:341)
at
com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:494)
at
com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:328)
at
com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:256)
at
javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:110)
at
org.apache.nifi.cluster.protocol.jaxb.JaxbProtocolContext$1.marshal(JaxbProtocolContext.java:75)
at
org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:182)
at
org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
--
Sanjeet Kumar Rath,
mob- +91 8777577470
