Sent too soon.
Hi Surneni, This looks to be an issue with your network setup. The address or hostname known to a Drillbit about another Drillbit is not resolved correctly while trying to make connection. Nodes with failure: drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt Can you please verify if above 2 nodes can ping each other using the hostname? If not may be you can try to update the /etc/hosts file to provide a mapping between hostname and corresponding IP address. with org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection. Failure type CONNECTION.. 2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] Thanks, Sorabh ________________________________ From: Sorabh Hamirwasia Sent: Wednesday, April 11, 2018 9:04 AM To: user@drill.apache.org Subject: Re: Apache Drill- Error setting up remote intermediate fragment execution Hi Surneni, This looks to be an issue with your network setup. The address or hostname known to a Drillbit about another Drillbit is not resolved correctly while trying to make connection. Thanks, Sorabh ________________________________ From: Surneni Tilak <surneni.ti...@ericsson.com> Sent: Wednesday, April 11, 2018 8:30 AM To: user@drill.apache.org Subject: FW: Apache Drill- Error setting up remote intermediate fragment execution Hi Team, I have successfully launched a Drill cluster on 3 docker containers with Zoo keeper Quorum and accessing data from a remote node on which Drill is not installed . I am able to list all the drill bits from each node and run queries on a single drill bit , But I am facing issue when I am trying to run a query which distributes the query to the other drill bits by creating the intermediate fragments. I have gone through the documentation but It couldn't help. I got the below error message in Drill shell and the detailed error log mentioned below. Please help me on resolving this. Error in Drill Shell: Error: CONNECTION ERROR: Error setting up remote intermediate fragment execution Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt [Error Id: 5d701a0d-988b-4845-83f6-b029ad6825a8 on drillmaster-76d6885c59-wpksz:31010] (state=,code=0) Error Log: 2018-04-11 07:26:17,391 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,401 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatus es() took 0 ms, numFiles: 1 2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Execute d 1 out of 1 using 1 threads. Time: 2ms total, 2.798853ms avg, 2ms max. 2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Execute d 1 out of 1 using 1 threads. Earliest start: 0.699000 ?s, Latest start: 0.699000 ?s, Average start: 0.699000 ?s . 2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pr uning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule 2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - No partition columns a re projected from the scan..continue. Total pruning elapsed time: 0 ms 2018-04-11 07:26:17,611 [BitServer-3] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89) [drill- rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76) [drill- rpc-1.12.0.jar:1.12.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Fina l] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) [netty-transport-4.0.48.Final.ja r:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.F inal.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [netty-transport-4.0.48.Final.ja r:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.F inal.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.F inal.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970) [netty-transport-4.0.48.Final.jar:4.0.48.Fina l] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [netty-common-4.0.48.Final. at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [netty-common-4.0.48.Final. jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4. 0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243) ~[netty-transport-4.0.48.Final.jar:4.0.48.Fi nal] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] ... 17 common frames omitted 2018-04-11 07:26:17,693 [BitServer-4] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89) [drill- rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76) [drill- rpc-1.12.0.jar:1.12.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Fina l] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) [netty-transport-4.0.48.Final.ja r:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.F inal.jar:4.0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243) ~[netty-transport-4.0.48.Final.jar:4.0.48.Fi nal] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] ... 17 common frames omitted 2018-04-11 07:26:17,694 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.drill.exec.work.foreman.Foreman - User Error Occurred: E rror setting up remote intermediate fragment execution (Command failed while establishing connection. Failure type CONNECTION.) org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error setting up remote intermediate fragment execution Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt [Error Id: 7e6e629d-1558-4a76-b9a9-6d34f86015a2 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) ~[drill-common-1.12.0.jar:1.12.0] at org.apache.drill.exec.work.foreman.Foreman.scheduleRemoteIntermediateFragments(Foreman.java:1126) [drill-java-exec-1.12.0.jar: 1.12.0] at org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1217) [drill-java-exec-1.12.0.jar:1.12.0] at org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454) [drill-java-exec-1.12.0.jar:1.12.0] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1019) [drill-java-exec-1.12.0.jar:1.12.0] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:289) [drill-java-exec-1.12.0.jar:1.12.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection. Failure type CONNECTION. at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66) ~[drill-rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152) ~ [drill-rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:118) ~[dril l-rpc-1.12.0.jar:1.12.0] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) ~[netty-transport-4.0.48.Final. jar:4.0.48.Final] ... 17 common frames omitted 2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State change requested AWAITING_ALLOCATION --> CANCELLATION_REQUESTED 2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State to repor t: CANCELLATION_REQUESTED 2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State change requested CANCELLATION_REQUESTED --> FINISHED 2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0: State to repor t: CANCELLED 2018-04-11 07:26:17,703 [BitServer-5] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89) [drill- rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76) [drill- rpc-1.12.0.jar:1.12.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Fina l] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) [netty-transport-4.0.48.Final.ja r:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.F inal.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.j ar:4.0.48.Final] ... 17 common frames omitted 2018-04-11 07:26:17,704 [BitServer-5] ERROR o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL fragment query_id { part1: 2680283233054685576 part2: 2356620362624167161 } major_fragment_id: 1 minor_fragment_id: 0 on endpoint address: "drillslave-7dccb67657-9qv4l" user_port: 31010 control_port: 31011 data_port: 31012 version: "1.12.0" state: ONLINE with org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection. Failure type CONNECTION.. 2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89) [drill- rpc-1.12.0.jar:1.12.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76) [drill- rpc-1.12.0.jar:1.12.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Fina l] Please respond back ASAP with your findings. Best regards, _________________________ Tilak