Re: Configure Impala for Kudu on Separate Cluster

Tue, 15 Aug 2017 22:45:09 -0700 

Ben,
As Todd mentioned, it might be some network connectivity problem. I would suspect some issues with connectivity between the node where the Impala shell is running and the Kudu master node. 


To start troubleshooting, I would verify that the node where you run the 
Impala shell (that's 172.35.120.191, right?) can establish a TCP 
connection to the master RPC end-point.  E.g., try to run from the 
command shell at 172.35.120.191:


  telnet 172.35.121.101 7051

Would it succeed?


Also, if running multi-master Kudu cluster, it might happen that masters 
cannot communicate with each other.  To troubleshoot that, I would try 
to establish a TCP connection to the RPC end-point of the master at one 
node from another master node.  E.g., if using telnet, from 
<master_node_0>, in the command-line shell:


  telnet <master_node_1> 7051

  (just substitute <master_node_0> and <master_node_1> with appropriate 
hostnames/IP addresses).




Best regards,

Alexey


On 8/15/17 9:53 PM, Benjamin Kim wrote:

Todd,


Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master 
config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 
<http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no 
leader. Exceptions received: 
org.apache.kudu.client.RecoverableException: [Peer Kudu Master - 
prod-dc1-datanode151.pdc1i.gradientx.com:7051 
<http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection 
reset on [id: 0x6232f33f, /172.35.120.191:47848 
<http://172.35.120.191:47848/> :> /172.35.121.101:7051 
<http://172.35.121.101:7051/>]



We got this error trying to use Kudu from within the cluster. Do you 
know what this means?


Cheers,
Ben



On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <t...@cloudera.com 
<mailto:t...@cloudera.com>> wrote:


    Is there a possibility that the remote node (prod-dc1-datanode151)
    is firewalled off from whatever host you are submitting the query
    to? The error message is admittedly pretty bad, but it basically
    means it's getting "connection refused", indicating that either
    there is no master running on that host or it has been blocked (eg
    an iptables REJECT rule)

    -Todd

    On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com
    <mailto:bbuil...@gmail.com>> wrote:

        Hi Todd,

        I tried to create a Kudu table using impala shell, and I got
        this error.

        create table my_first_table
        (
          id bigint,
          name string,
          primary key(id)
        )
        partition by hash partitions 16
        stored as kudu;
        Query: create table my_first_table
        (
          id bigint,
          name string,
          primary key(id)
        )
        partition by hash partitions 16
        stored as kudu
        ERROR: ImpalaRuntimeException: Error creating Kudu table
        'impala::default.my_first_table'
        CAUSED BY: NonRecoverableException: Too many attempts:
        KuduRpc(method=ListTables, tablet=null, attempt=101,
        DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms]
        querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to
        server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [2ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [5ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [21ms] querying master, [22ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [24ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [26ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [41ms] querying master, [41ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [43ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [46ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [62ms] querying master, [62ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [64ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [66ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [81ms] querying master, [81ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [84ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [86ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [122ms] querying master, [122ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [123ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [124ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [161ms] querying master, [162ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [163ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [166ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [302ms] querying master, [302ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [304ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [306ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [382ms] querying master, [382ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [384ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [386ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [622ms] querying master, [622ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [623ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [626ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [701ms] querying master, [702ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [703ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [706ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [2062ms] querying master, [2062ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [2065ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [2068ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [4661ms] querying master, [4662ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [4663ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [4666ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [6961ms] querying master, [6962ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [6964ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [6967ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [9102ms] querying master, [9102ms] Sub rpc:
        ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [9104ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [9106ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [12001ms] querying master, [12002ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [12003ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [12006ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [15822ms] querying master, [15822ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [15824ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [15826ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [16002ms] querying master, [16002ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [16004ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [16006ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [18381ms] querying master, [18381ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [18383ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [18386ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [18821ms] querying master, [18822ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [18823ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [18826ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [19262ms] querying master, [19262ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [19263ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [19267ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [23082ms] querying master, [23082ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [23083ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [23086ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [26062ms] querying master, [26063ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [26064ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [26067ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [29721ms] querying master, [29722ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [29723ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [29726ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [30141ms] querying master, [30143ms] Sub
        rpc: ConnectToMaster sending RPC to server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>,
        [30144ms] Sub rpc: ConnectToMaster received from server
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>
        response Network error: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [30147ms] delaying RPC due to Service
        unavailable: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed, [33361ms] trace too long, truncated)
        CAUSED BY: NoLeaderFoundException: Master config
        (prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no
        leader. Exceptions received:
        org.apache.kudu.client.RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed
        CAUSED BY: RecoverableException: [Peer
        master-prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>]
        Connection closed

        I also added to Impala Command Line Argument Advanced
        Configuration Snippet (Safety Valve) the setting:
        -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051
        <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>

        And to Impala Service Environment Advanced Configuration
        Snippet (Safety Valve) the setting:
        IMPALA_KUDU=1

        I don't think these are correct because it's not working.

        Cheers,
        Ben


        On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com
        <mailto:t...@cloudera.com>> wrote:

            Hi Ben,

            What error are you getting? The Impala shell shouldn't be
            relevant here -- it's just submitting queries to the
            Impala daemons themselves.

            This deployment scenario hasn't been tested much as far as
            I know, so I wouldn't be surprised if there are issues
            with scheduling fragments given lack of locality, etc, but
            I would expect it to at least "basically work". (I've done
            it once or twice to copy data from one cluster to another)

            -Todd

            On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim
            <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote:

                Can someone help me with configuring Impala using
                Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I
                cannot get it to connect using impala shell.

                Cheers,
                Ben





            -- 
            Todd Lipcon

            Software Engineer, Cloudera





    -- 
    Todd Lipcon

    Software Engineer, Cloudera

























					Reply via email to