Hi all,
We are running into a small issue when using Drill to query S3. We could use your input to help resolve this. Issue: - While querying using the S3 Storage plugin, when a query is of the form ’SELECT col1, col2 from s3.root.`/dir/file1.csv` UNION SELECT col1, col2 from s3.root.`/dir/file2.csv`’, (querying single files at a time works) we get this error: CONNECTION ERROR: Error setting up remote intermediate fragment execution Nodes with failures drill-c9d87fd6f-7pjxn, drill-c9d87fd6f-22kbn (Complete error from log given below) Steps we took: - Error seems to be from the FragmentsRunner.scheduleIntermediateFragments(final Multimap<DrillbitEndpoint, PlanFragment> intermediateFragmentMap) method. The Nodes with failures are ‘endpoint.getAddress()’ values - We tried pinging each pod from within the other using their IP address and it worked. Deployment in cluster: - We are using Drill 1.13.0 (Docker hub image from https://hub.docker.com/r/drill/apache-drill-centos/). Also tested this on the Drill 1.14.0 image and getting same results. - For zookeeper, we deploy a Stateful Set in our namespace using the image: gcr.io/google_samples/k8szk:v2 Please let us know if there is any configuration needed to be able to query from S3? UNION queries work fine while working with mounted Network File System directories. Thanks and Regards, Vedant Naik Complete error from logs: 2018-08-21 15:23:36,660 [2483d188-a545-1971-fbd5-e96efa38bce6:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 0.918000 ?s, Latest start: 0.918000 ?s, Average start: 0.918000 ?s . 2018-08-21 15:23:41,284 [BitServer-4] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:90) [drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77) [drill-rpc-1.13.0.jar:1.13.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) [netty-transport-4.0.48.Final.jar:4.0.48.Final] ... 17 common frames omitted 2018-08-21 15:23:41,465 [BitServer-7] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:90) [drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77) [drill-rpc-1.13.0.jar:1.13.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) [netty-transport-4.0.48.Final.jar:4.0.48.Final] ... 17 common frames omitted 2018-08-21 15:23:41,470 [2483d188-a545-1971-fbd5-e96efa38bce6:foreman] INFO o.a.d.e.work.foreman.FragmentsRunner - User Error Occurred: Error setting up remote intermediate fragment execution (Command failed while establishing connection. Failure type CONNECTION.) org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error setting up remote intermediate fragment execution Nodes with failures drill-c9d87fd6f-7pjxn, drill-c9d87fd6f-22kbn [Error Id: 921b8e47-3e25-4100-b7ae-202faa75dd0d ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.FragmentsRunner.scheduleRemoteIntermediateFragments(FragmentsRunner.java:311) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.FragmentsRunner.setupNonRootFragments(FragmentsRunner.java:196) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.FragmentsRunner.submit(FragmentsRunner.java:112) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.Foreman.runFragments(Foreman.java:487) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.Foreman.startQueryProcessing(Foreman.java:462) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:410) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:569) [drill-java-exec-1.13.0.jar:1.13.0] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264) [drill-java-exec-1.13.0.jar:1.13.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection. Failure type CONNECTION. at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66) ~[drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152) ~[drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:119) ~[drill-rpc-1.13.0.jar:1.13.0] at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77) ~[drill-rpc-1.13.0.jar:1.13.0] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] ... 1 common frames omitted Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161] at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205) ~[netty-transport-4.0.48.Final.jar:4.0.48.Final] ... 17 common frames omitted -- *Kind Regards,* *Vedant Naik.*
