[ https://issues.apache.org/jira/browse/STORM-128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Kellogg updated STORM-128: ------------------------------- Component/s: storm-core > Topology fails to start if a configured DRPC server is down > ----------------------------------------------------------- > > Key: STORM-128 > URL: https://issues.apache.org/jira/browse/STORM-128 > Project: Apache Storm > Issue Type: Bug > Components: storm-core > Reporter: James Xu > Priority: Minor > > https://github.com/nathanmarz/storm/issues/696 > In our environment we have 3 DRPC servers running. This was done mainly for > availability and capacity. However, we noticed that when even one of these > servers is down, topologies fail to start with the following exception: > java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: > java.net.NoRouteToHostException: No route to host > at backtype.storm.drpc.DRPCInvocationsClient.(DRPCInvocationsClient.java:23) > at backtype.storm.drpc.DRPCSpout.open(DRPCSpout.java:65) > at > storm.trident.spout.RichSpoutBatchTriggerer.open(RichSpoutBatchTriggerer.java:41) > at backtype.storm.daemon.executor$fn__3985$fn__3997.invoke(executor.clj:460) > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375) > at clojure.lang.AFn.run(AFn.java:24) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.apache.thrift7.transport.TTransportException: > java.net.NoRouteToHostException: No route to host > at org.apache.thrift7.transport.TSocket.open(TSocket.java:183) > at > org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) > at > backtype.storm.drpc.DRPCInvocationsClient.connect(DRPCInvocationsClient.java:30) > at backtype.storm.drpc.DRPCInvocationsClient.(DRPCInvocationsClient.java:21) > ... 6 more > Caused by: java.net.NoRouteToHostException: No route to host > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) > at java.net.Socket.connect(Socket.java:579) > at org.apache.thrift7.transport.TSocket.open(TSocket.java:178) > ... 9 more > I was wondering if it makes sense to make Storm handle this gracefully > instead of failing fast. Otherwise, the DRPC servers become a SPOF. > If the topologies are already running the topology usually just logs an error > message and continues. > ---------- > dkador: +1 on figuring out how to make the DRPC stuff not a SOP. I'd be happy > to look into it myself but not sure where to start. Any guidance? > ---------- > rijuk: For reference, the stack trace I see when a DRPC server goes down > while a topology is running is the following. In this case, the topology > continues to function normally. > [backtype.storm.drpc.DRPCSpout Thread-65]: Failed to fetch DRPC result from > DRPC server > org.apache.thrift7.transport.TTransportException: java.net.ConnectException: > Connection refused > at org.apache.thrift7.transport.TSocket.open(TSocket.java:183) > at > org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) > at > backtype.storm.drpc.DRPCInvocationsClient.connect(DRPCInvocationsClient.java:30) > at > backtype.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:53) > at backtype.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:89) > at > storm.trident.spout.RichSpoutBatchTriggerer.nextTuple(RichSpoutBatchTriggerer.java:68) > at > backtype.storm.daemon.executor$fn__3985$fn__3997$fn__4026.invoke(executor.clj:502) > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) > at clojure.lang.AFn.run(AFn.java:24) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) > at java.net.Socket.connect(Socket.java:579) > at org.apache.thrift7.transport.TSocket.open(TSocket.java:178) > ... 9 more > In this case I'd the host up, but the DRPC server process was down. Hence the > ConnectException. But, the behavior is the same even when the host is > unreachable, except for the Exception type. > @dkador, I'm not sure what the right solution is. One naive solution I can > think of is to make DRPCInvocationsClient constructor rethrow TException > instead of throwing a RuntimeException. Obviously, you'll have to make sure > that all callers of this higher up in the stack handle this exception > properly. > Actually, on second thoughts that's not a good idea. You probably still want > the DRPCInvocationsClient object to be constructed. So, maybe you can log an > error and just eat that exception. All other methods in that class call > "connect" if necessary anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332)