Hi, I have a storm 2.4.0 cluster with all the storm components such as Nimbus, UI, supervisors, and drpc. Recently we moved to a new GCP Kubernetes cluster where both traffic was coming from both IPv4 and IPv6 addresses. Previously, we were on a hashicorp nomad cluster where only IPv4 was supported.
Since the move to GCP Kubernetes (v1.23.x), I started to see supervisor is unable to communicate with drpc on port 3773. This is the error snippet from one of the topology logs inside a supervisor. 2023-06-01 19:02:10.136 o.a.s.d.DRPCSpout Thread-28-spout-executor[194, 194] [ERROR] Failed to fetch DRPC request from DRPC server [search-storm-drpc-4.search-storm-drpc:3773] org.apache.storm.thrift.transport.TTransportException: Socket is closed by peer. at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:130) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-shaded-deps-2.4.0.jar:2.4.0] at org.apache.storm.generated.DistributedRPCInvocations$Client.recv_fetchRequest(DistributedRPCInvocations.java:114) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.generated.DistributedRPCInvocations$Client.fetchRequest(DistributedRPCInvocations.java:101) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:84) ~[storm-client-2.4.0.jar:2.4.0] at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:175) [storm-client-2.4.0.jar:2.4.0] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:187) [storm-client-2.4.0.jar:2.4.0] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:153) [storm-client-2.4.0.jar:2.4.0] at org.apache.storm.utils.Utils$1.run(Utils.java:396) [storm-client-2.4.0.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] On further debugging, I see all storm-drpc ports (3772, 3773, 3774) are listening to IPv6 addresses. [image: image.png] I tried adding the JVM flag *-Djava.net.preferIPv4Stack=true *mentioned here https://storm.apache.org/releases/2.4.0/Troubleshooting.html but that also did not help. The weird thing is, telnet to the DRPC DNS names and ports work from all pods, but still, I see socket errors in supervisor pods while talking to DRPC. Any inputs on any difficulties or workaround to run Storm 2.4.0 in IPv6 addresses or make it successfully run by making ports listen to IPv4 addresses? Regards, Ash