Hello Aishwarya, Are you using a helm chart for this deploy and/or k8s
deployment/service files that you can share? I don't believe Storm has been
extensively tested in IPv6.
Thanks--Bipin
On Thursday, June 1, 2023 at 12:20:45 PM PDT, Aishwarya Soni
<[email protected]> wrote:
Hi,
I have a storm 2.4.0 cluster with all the storm components such as Nimbus, UI,
supervisors, and drpc. Recently we moved to a new GCP Kubernetes cluster where
both traffic was coming from both IPv4 and IPv6 addresses. Previously, we were
on a hashicorp nomad cluster where only IPv4 was supported.
Since the move to GCP Kubernetes (v1.23.x), I started to see supervisor is
unable to communicate with drpc on port 3773. This is the error snippet from
one of the topology logs inside a supervisor.
2023-06-01 19:02:10.136 o.a.s.d.DRPCSpout Thread-28-spout-executor[194, 194]
[ERROR] Failed to fetch DRPC request from DRPC server
[search-storm-drpc-4.search-storm-drpc:3773]
org.apache.storm.thrift.transport.TTransportException: Socket is closed by peer.
at
org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:130)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.generated.DistributedRPCInvocations$Client.recv_fetchRequest(DistributedRPCInvocations.java:114)
~[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.generated.DistributedRPCInvocations$Client.fetchRequest(DistributedRPCInvocations.java:101)
~[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:84)
~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:175)
[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:187)
[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:153)
[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.utils.Utils$1.run(Utils.java:396)
[storm-client-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
On further debugging, I see all storm-drpc ports (3772, 3773, 3774) are
listening to IPv6 addresses.
I tried adding the JVM flag -Djava.net.preferIPv4Stack=true mentioned here
https://storm.apache.org/releases/2.4.0/Troubleshooting.html but that also did
not help.
The weird thing is, telnet to the DRPC DNS names and ports work from all pods,
but still, I see socket errors in supervisor pods while talking to DRPC.
Any inputs on any difficulties or workaround to run Storm 2.4.0 in IPv6
addresses or make it successfully run by making ports listen to IPv4 addresses?
Regards,
Ash
