Hello Adam,
I had encountered a similar issue before - DRILL-3075
<https://issues.apache.org/jira/browse/DRILL-3075>. As a workaround, the
following changes helped me resolve the issue:
(1) Update /etc/hosts on each of the 3 nodes with the hostnames and IP
addresses of each node.
(2) If that doesn't resolve the issue, udpate your connection string to
connect directly to a specific drillbit (jdbc:drill:drillbit=<IP Addr>)
Let us know if that helps.
Regards,
Abhishek
On Wed, Jan 20, 2016 at 12:25 PM, Adam Brinckman <
adam.brinck...@aunalytics.com> wrote:
> I am trying to setup a drill cluster running 3 vm nodes running CentOS 7. I
> have successfully installed a small zookeeper ensemble on the three nodes
> and then installed drill as per the documentation. I am having trouble
> connecting to the other drillbits when running drill-conf: However, this
> issue is intermittent. I can eventually connect if I spam drill-conf.
>
> Error: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.RpcException: CONNECTION :
> java.nio.channels.UnresolvedAddressException (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill:
> org.apache.drill.exec.rpc.RpcException: CONNECTION :
> java.nio.channels.UnresolvedAddressException
> at
>
> org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:159)
> at
>
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:64)
> at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at
>
> net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.exec.rpc.RpcException: CONNECTION :
> java.nio.channels.UnresolvedAddressException
> at
>
> org.apache.drill.exec.client.DrillClient$FutureHandler.connectionFailed(DrillClient.java:448)
> at
>
> org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:237)
> at
>
> org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:200)
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
>
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
> at
>
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.connect(AbstractEpollStreamChannel.java:482)
> at
>
> io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1089)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:543)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:528)
> at
>
> io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:543)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:528)
> at
> io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:543)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:528)
> at
>
> io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:510)
> at
>
> io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:909)
> at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:203)
> at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:165)
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
> at
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.nio.channels.UnresolvedAddressException
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47)
> at
>
> org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:213)
> ... 22 more
> Caused by: java.nio.channels.UnresolvedAddressException
> at
>
> io.netty.channel.epoll.AbstractEpollChannel.checkResolvable(AbstractEpollChannel.java:221)
> at
>
> io.netty.channel.epoll.EpollSocketChannel.doConnect(EpollSocketChannel.java:183)
> at
>
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.connect(AbstractEpollStreamChannel.java:445)
> ... 17 more
> apache drill 1.4.0
> "the only truly happy people are children, the creative minority and drill
> users"
> 0: jdbc:drill:>
>
>
>
> As I said, I will eventually connect only after spamming drill-conf a few
> times.
>
> In addition to this failure, I am unable to run broadcast joins when I
> point my drill cluster to an external HDH datasource. This failure happens
> 100% of the time:
>
> Error: CONNECTION ERROR: Error setting up remote intermediate fragment
> execution
>
> Nodes with failures dev-drill-node3, dev-drill-node2
>
> [Error Id: 13a566d2-b4c0-4be9-8f04-da77c4dfef23 on dev-drill-node1:31010]
> (state=,code=0)
>
> Doing a grep on that error id I can see this error message:
>
> 2016-01-20 15:17:48,048 [BitServer-1] ERROR
> o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
> fragment query_id {
> part1: 2981405579761470863
> part2: 8997799343876536749
> }
> major_fragment_id: 1
> minor_fragment_id: 1
> on endpoint address: "dev-drill-node3"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> with org.apache.drill.exec.rpc.RpcException: Command failed while
> establishing connection. Failure type CONNECTION..
> 2016-01-20 15:17:48,048 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:0:0: State change requested
> AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,048 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:0:0: State to report:
> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,048 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:0:0: State change requested
> CANCELLATION_REQUESTED --> FINISHED
> 2016-01-20 15:17:48,048 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:0:0: State to report: CANCELLED
> 2016-01-20 15:17:48,049 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:2:0: State change requested
> AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,049 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:2:0: State to report:
> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,049 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:2:0: State change requested
> CANCELLATION_REQUESTED --> FINISHED
> 2016-01-20 15:17:48,049 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:2:0: State to report: CANCELLED
> 2016-01-20 15:17:48,050 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:1:0: State change requested
> AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,050 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:1:0: State to report:
> CANCELLATION_REQUESTED
> 2016-01-20 15:17:48,050 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:1:0: State change requested
> CANCELLATION_REQUESTED --> FINISHED
> 2016-01-20 15:17:48,050 [CONTROL-rpc-event-queue] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29601494-210c-cd8f-7cde-9ad4b400f9ad:1:0: State to report: CANCELLED
> 2016-01-20 15:17:48,054 [BitServer-1] ERROR
> o.apache.drill.exec.rpc.BasicClient - Failed to establish connection
> java.util.concurrent.ExecutionException:
> java.nio.channels.UnresolvedAddressException
>
>
> I have spent the last week and half pulling my hair out. I've combed
> through the documentations but cannot find any information about may
> causing this. I am at this point in desperate need for help on this
> problem. Any insight into what may be causing this would be met with
> extreme gratitude. If I haven't provided enough documentation please
> contact me as soon as possible.
>
> Thank You
>
