Hi Good People!
I’m trying to update my cluster running Storm 0.10.0 with DRPC to Storm 1.0.0.
I’ve updated all machines with latest version of Storm, changed storm.yml
configurations (e.g. nimbus.host -> nimbus.seeds), changed dependencies in
topology. When I starting nimbus, supervisors, ui, drpc, logviewers everything
looks good, I’m able to see UI, logs, configuration and submit topologies.
But once I submit DRPC topology – it’s logs full of following exceptions:
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:129)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:210)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.daemon.executor$fn__8158$fn__8173$fn__8204.invoke(executor.clj:647)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.0.jar:1.0.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_74]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_74]
at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:127)
~[storm-core-1.0.0.jar:1.0.0]
... 5 more
Caused by: java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at
org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_74]
... 1 more
Caused by: org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_74]
... 1 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_74]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_74]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
~[?:1.8.0_74]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_74]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_74]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_74]
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_74]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_74]
... 1 more
2016-04-18 18:10:19.579 o.a.s.util [ERROR] Async loop died!
It seems DRPCSpout is unable to connect to DRPC server in some reasons.
Everything is fine with Storm 0.10.0
Supervisor’s configuration is correct – there is valid ‘drpc.servers’ entry.
Zookeeper and Nimbus are running
I’m able to telnet DRPC server 3772, 3774, 3773 ports from any supervisor.
DRPC server is up and running, it is able to handle request (with timeout of
course), here is the log:
2016-04-18 18:00:20.444 o.a.s.d.drpc [INFO] Starting drpc server for storm
version '1.0.0'
2016-04-18 18:00:20.619 o.a.s.d.drpc [INFO] Starting Distributed RPC servers...
2016-04-18 18:00:20.692 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
2016-04-18 18:00:20.717 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
2016-04-18 18:00:20.728 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
SelectChannelConnector@0.0.0.0:3774
2016-04-18 18:00:20.729 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter
plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2016-04-18 18:00:20.730 o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing...
2016-04-18 18:00:20.737 o.a.s.d.common [INFO] Started statistics report
plugin...
2016-04-18 18:09:49.790 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR]
Read an invalid frame size of -786435. Are you using TFramedTransport on the
client side?
2016-04-18 18:26:10.413 o.a.s.d.drpc [INFO] Starting drpc server for storm
version '1.0.0'
2016-04-18 18:26:10.743 o.a.s.d.drpc [INFO] Starting Distributed RPC servers...
2016-04-18 18:26:10.852 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
2016-04-18 18:26:10.896 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
2016-04-18 18:26:10.921 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
SelectChannelConnector@0.0.0.0:3774
2016-04-18 18:26:10.922 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter
plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2016-04-18 18:26:10.924 o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing...
2016-04-18 18:26:10.935 o.a.s.d.common [INFO] Started statistics report
plugin...
Please help
Sincerely,
Viktor
