Try setting these properties in your profile
"fs.s3a.multipart.size": "10485760",
"fs.s3a.multipart.threshold": "104857600",
"fs.s3a.connection.maximum": "10000"
On Wed, Aug 22, 2018 at 7:24 AM Vedant Naik <naik.ved...@gmail.com> wrote:
> Hi all,
>
>
> We are running into a small issue when using Drill to query S3. We could
> use your input to help resolve this.
>
>
> Issue:
>
> - While querying using the S3 Storage plugin, when a query is of the
> form ’SELECT col1, col2 from s3.root.`/dir/file1.csv` UNION SELECT col1,
> col2 from s3.root.`/dir/file2.csv`’, (querying single files at a time
> works) we get this error:
>
> CONNECTION ERROR: Error setting up remote intermediate fragment execution
>
> Nodes with failures drill-c9d87fd6f-7pjxn, drill-c9d87fd6f-22kbn
>
> (Complete error from log given below)
>
>
> Steps we took:
>
> - Error seems to be from the
> FragmentsRunner.scheduleIntermediateFragments(final
> Multimap<DrillbitEndpoint,
> PlanFragment> intermediateFragmentMap) method. The Nodes with failures
> are
> ‘endpoint.getAddress()’ values
> - We tried pinging each pod from within the other using their IP address
> and it worked.
>
>
> Deployment in cluster:
>
> - We are using Drill 1.13.0 (Docker hub image from
> https://hub.docker.com/r/drill/apache-drill-centos/). Also tested this
> on the Drill 1.14.0 image and getting same results.
> - For zookeeper, we deploy a Stateful Set in our namespace using the
> image: gcr.io/google_samples/k8szk:v2
>
>
> Please let us know if there is any configuration needed to be able to query
> from S3? UNION queries work fine while working with mounted Network File
> System directories.
>
>
> Thanks and Regards,
>
> Vedant Naik
>
>
> Complete error from logs:
>
> 2018-08-21 15:23:36,660 [2483d188-a545-1971-fbd5-e96efa38bce6:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1
> using 1 threads. Earliest start: 0.918000 ?s, Latest start: 0.918000 ?s,
> Average start: 0.918000 ?s .
>
> 2018-08-21 15:23:41,284 [BitServer-4] ERROR
> o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection
>
> java.util.concurrent.ExecutionException:
> java.nio.channels.UnresolvedAddressException
>
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:90)
> [drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77)
> [drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
>
> Caused by: java.nio.channels.UnresolvedAddressException: null
>
> at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
>
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> ... 17 common frames omitted
>
> 2018-08-21 15:23:41,465 [BitServer-7] ERROR
> o.a.d.e.rpc.ConnectionMultiListener - Failed to establish connection
>
> java.util.concurrent.ExecutionException:
> java.nio.channels.UnresolvedAddressException
>
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:90)
> [drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77)
> [drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
>
> Caused by: java.nio.channels.UnresolvedAddressException: null
>
> at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
>
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
> [netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> ... 17 common frames omitted
>
> 2018-08-21 15:23:41,470 [2483d188-a545-1971-fbd5-e96efa38bce6:foreman] INFO
> o.a.d.e.work.foreman.FragmentsRunner - User Error Occurred: Error setting
> up remote intermediate fragment execution (Command failed while
> establishing connection. Failure type CONNECTION.)
>
> org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error
> setting up remote intermediate fragment execution
>
>
> Nodes with failures drill-c9d87fd6f-7pjxn, drill-c9d87fd6f-22kbn
>
>
> [Error Id: 921b8e47-3e25-4100-b7ae-202faa75dd0d ]
>
> at
>
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work
> .foreman.FragmentsRunner.scheduleRemoteIntermediateFragments(FragmentsRunner.java:311)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work
> .foreman.FragmentsRunner.setupNonRootFragments(FragmentsRunner.java:196)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work
> .foreman.FragmentsRunner.submit(FragmentsRunner.java:112)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work.foreman.Foreman.runFragments(Foreman.java:487)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work
> .foreman.Foreman.startQueryProcessing(Foreman.java:462)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
> org.apache.drill.exec.work
> .foreman.Foreman.runPhysicalPlan(Foreman.java:410)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:569)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> [drill-java-exec-1.13.0.jar:1.13.0]
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
>
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
>
> Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while
> establishing connection. Failure type CONNECTION.
>
> at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> ~[drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> ~[drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152)
> ~[drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:119)
> ~[drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77)
> ~[drill-rpc-1.13.0.jar:1.13.0]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
> io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> ... 1 common frames omitted
>
> Caused by: java.nio.channels.UnresolvedAddressException: null
>
> at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
>
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.8.0_161]
>
> at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
> ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> at
>
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
> ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>
> ... 17 common frames omitted
> --
> *Kind Regards,*
> *Vedant Naik.*
>
--
Nitin Pawar
