Re: Configure Impala for Kudu on Separate Cluster

[Alexey Serbin]

Ben,

As Todd mentioned, it might be some network connectivity problem. I 
would suspect some issues with connectivity between the node where the 
Impala shell is running and the Kudu master node.


To start troubleshooting, I would verify that the node where you run the 
Impala shell (that's 172.35.120.191, right?) can establish a TCP 
connection to the master RPC end-point.  E.g., try to run from the 
command shell at 172.35.120.191:


  telnet 172.35.121.101 7051

Would it succeed?

Also, if running multi-master Kudu cluster, it might happen that masters 
cannot communicate with each other.  To troubleshoot that, I would try 
to establish a TCP connection to the RPC end-point of the master at one 
node from another master node.  E.g., if using telnet, from 
, in the command-line shell:


  telnet  7051
  (just substitute  and  with appropriate 
hostnames/IP addresses).




Best regards,

Alexey


On 8/15/17 9:53 PM, Benjamin Kim wrote:

Todd,

Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master 
config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 
) has no 
leader. Exceptions received: 
org.apache.kudu.client.RecoverableException: [Peer Kudu Master - 
prod-dc1-datanode151.pdc1i.gradientx.com:7051 
] Connection 
reset on [id: 0x6232f33f, /172.35.120.191:47848 
 :> /172.35.121.101:7051 
]


We got this error trying to use Kudu from within the cluster. Do you 
know what this means?


Cheers,
Ben


On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon > wrote:


Is there a possibility that the remote node (prod-dc1-datanode151)
is firewalled off from whatever host you are submitting the query
to? The error message is admittedly pretty bad, but it basically
means it's getting "connection refused", indicating that either
there is no master running on that host or it has been blocked (eg
an iptables REJECT rule)

-Todd

On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim > wrote:

Hi Todd,

I tried to create a Kudu table using impala shell, and I got
this error.

create table my_first_table
(
  id bigint,
  name string,
  primary key(id)
)
partition by hash partitions 16
stored as kudu;
Query: create table my_first_table
(
  id bigint,
  name string,
  primary key(id)
)
partition by hash partitions 16
stored as kudu
ERROR: ImpalaRuntimeException: Error creating Kudu table
'impala::default.my_first_table'
CAUSED BY: NonRecoverableException: Too many attempts:
KuduRpc(method=ListTables, tablet=null, attempt=101,
DeadlineTracker(timeout=18, elapsed=178226), Traces: [0ms]
querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to
server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
,
[2ms] Sub rpc: ConnectToMaster received from server
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051

response Network error: [Peer
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
]
Connection closed, [5ms] delaying RPC due to Service
unavailable: Master config
(prod-dc1-datanode151.pdc1i.gradientx.com:7051
) has no
leader. Exceptions received:
org.apache.kudu.client.RecoverableException: [Peer
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
]
Connection closed, [21ms] querying master, [22ms] Sub rpc:
ConnectToMaster sending RPC to server
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
,
[24ms] Sub rpc: ConnectToMaster received from server
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051

response Network error: [Peer
master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
]
Connection closed, [26ms] delaying RPC due to Service
unavailable: Master config
(prod-dc1-datanode151.pdc1i.gradientx.com:7051
) has no
leader. Exceptions received:
org.apache.kudu.client.RecoverableException: [Peer

Re: Configure Impala for Kudu on Separate Cluster

[Benjamin Kim]

Todd,

Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master
config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
Kudu Master - prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection
reset on [id: 0x6232f33f, /172.35.120.191:47848 :> /172.35.121.101:7051]

We got this error trying to use Kudu from within the cluster. Do you know
what this means?

Cheers,
Ben


On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon  wrote:

> Is there a possibility that the remote node (prod-dc1-datanode151) is
> firewalled off from whatever host you are submitting the query to? The
> error message is admittedly pretty bad, but it basically means it's getting
> "connection refused", indicating that either there is no master running on
> that host or it has been blocked (eg an iptables REJECT rule)
>
> -Todd
>
> On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim  wrote:
>
>> Hi Todd,
>>
>> I tried to create a Kudu table using impala shell, and I got this error.
>>
>> create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu;
>> Query: create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu
>> ERROR: ImpalaRuntimeException: Error creating Kudu table
>> 'impala::default.my_first_table'
>> CAUSED BY: NonRecoverableException: Too many attempts:
>> KuduRpc(method=ListTables, tablet=null, attempt=101,
>> DeadlineTracker(timeout=18, elapsed=178226), Traces: [0ms] querying
>> master, [1ms] Sub rpc: ConnectToMaster sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [5ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [21ms] querying master, [22ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [24ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [26ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [41ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [43ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [46ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [62ms] querying master, [62ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [64ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [66ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [81ms] querying master, [81ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [84ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [86ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [122ms] querying master, [122ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, 

Re: Configure Impala for Kudu on Separate Cluster

[Benjamin Kim]

Todd,

I'll check this with the systems team.

Cheers,
Ben


On Mon, Aug 14, 2017 at 10:40 PM Todd Lipcon  wrote:

> Is there a possibility that the remote node (prod-dc1-datanode151) is
> firewalled off from whatever host you are submitting the query to? The
> error message is admittedly pretty bad, but it basically means it's getting
> "connection refused", indicating that either there is no master running on
> that host or it has been blocked (eg an iptables REJECT rule)
>
> -Todd
>
> On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim  wrote:
>
>> Hi Todd,
>>
>> I tried to create a Kudu table using impala shell, and I got this error.
>>
>> create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu;
>> Query: create table my_first_table
>> (
>>   id bigint,
>>   name string,
>>   primary key(id)
>> )
>> partition by hash partitions 16
>> stored as kudu
>> ERROR: ImpalaRuntimeException: Error creating Kudu table
>> 'impala::default.my_first_table'
>> CAUSED BY: NonRecoverableException: Too many attempts:
>> KuduRpc(method=ListTables, tablet=null, attempt=101,
>> DeadlineTracker(timeout=18, elapsed=178226), Traces: [0ms] querying
>> master, [1ms] Sub rpc: ConnectToMaster sending RPC to server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2ms] Sub rpc:
>> ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [5ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [21ms] querying master, [22ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [24ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [26ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [41ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [43ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [46ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [62ms] querying master, [62ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [64ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [66ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [81ms] querying master, [81ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [84ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [86ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed,
>> [122ms] querying master, [122ms] Sub rpc: ConnectToMaster sending RPC to
>> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [123ms] Sub
>> rpc: ConnectToMaster received from server
>> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network
>> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051]
>> Connection closed, [124ms] delaying RPC due to Service unavailable: Master
>> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader.
>> Exceptions received: