Hi, I found 2 issues. First, network connection is blocked. I filed a request to open it. Second, Kudu 1.4 had problems registering any tablet server. So, I reverted back to 1.3.1. The error had to do security and how IP routing was not trusted.
Also, I installed Kudu onto another cluster where I knew the network would not be a problem, and it worked. Cheers, Ben On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin <aser...@cloudera.com> wrote: > Ben, > > As Todd mentioned, it might be some network connectivity problem. I > would suspect some issues with connectivity between the node where the > Impala shell is running and the Kudu master node. > > To start troubleshooting, I would verify that the node where you run the > Impala shell (that's 172.35.120.191, right?) can establish a TCP > connection to the master RPC end-point. E.g., try to run from the > command shell at 172.35.120.191: > > telnet 172.35.121.101 7051 > > Would it succeed? > > Also, if running multi-master Kudu cluster, it might happen that masters > cannot communicate with each other. To troubleshoot that, I would try > to establish a TCP connection to the RPC end-point of the master at one > node from another master node. E.g., if using telnet, from > <master_node_0>, in the command-line shell: > > telnet <master_node_1> 7051 > (just substitute <master_node_0> and <master_node_1> with appropriate > hostnames/IP addresses). > > > > Best regards, > > Alexey > > > On 8/15/17 9:53 PM, Benjamin Kim wrote: > > Todd, > > > > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master > > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer Kudu Master - > > prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection > > reset on [id: 0x6232f33f, /172.35.120.191:47848 > > <http://172.35.120.191:47848/> :> /172.35.121.101:7051 > > <http://172.35.121.101:7051/>] > > > > We got this error trying to use Kudu from within the cluster. Do you > > know what this means? > > > > Cheers, > > Ben > > > > > > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <t...@cloudera.com > > <mailto:t...@cloudera.com>> wrote: > > > > Is there a possibility that the remote node (prod-dc1-datanode151) > > is firewalled off from whatever host you are submitting the query > > to? The error message is admittedly pretty bad, but it basically > > means it's getting "connection refused", indicating that either > > there is no master running on that host or it has been blocked (eg > > an iptables REJECT rule) > > > > -Todd > > > > On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com > > <mailto:bbuil...@gmail.com>> wrote: > > > > Hi Todd, > > > > I tried to create a Kudu table using impala shell, and I got > > this error. > > > > create table my_first_table > > ( > > id bigint, > > name string, > > primary key(id) > > ) > > partition by hash partitions 16 > > stored as kudu; > > Query: create table my_first_table > > ( > > id bigint, > > name string, > > primary key(id) > > ) > > partition by hash partitions 16 > > stored as kudu > > ERROR: ImpalaRuntimeException: Error creating Kudu table > > 'impala::default.my_first_table' > > CAUSED BY: NonRecoverableException: Too many attempts: > > KuduRpc(method=ListTables, tablet=null, attempt=101, > > DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] > > querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to > > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [2ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [5ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [21ms] querying master, [22ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [24ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [26ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [41ms] querying master, [41ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [43ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [46ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [62ms] querying master, [62ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [64ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [66ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [81ms] querying master, [81ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [84ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [86ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [122ms] querying master, [122ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [123ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [124ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [161ms] querying master, [162ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [163ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [166ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [302ms] querying master, [302ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [304ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [306ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [382ms] querying master, [382ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [384ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [386ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [622ms] querying master, [622ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [623ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [626ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [701ms] querying master, [702ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [703ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [706ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [2062ms] querying master, [2062ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [2065ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [2068ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [4661ms] querying master, [4662ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [4663ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [4666ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [6961ms] querying master, [6962ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [6964ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [6967ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [9102ms] querying master, [9102ms] Sub rpc: > > ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [9104ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [9106ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [12001ms] querying master, [12002ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [12003ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [12006ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [15822ms] querying master, [15822ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [15824ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [15826ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [16002ms] querying master, [16002ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [16004ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [16006ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [18381ms] querying master, [18381ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [18383ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [18386ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [18821ms] querying master, [18822ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [18823ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [18826ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [19262ms] querying master, [19262ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [19263ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [19267ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [23082ms] querying master, [23082ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [23083ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [23086ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [26062ms] querying master, [26063ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [26064ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [26067ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [29721ms] querying master, [29722ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [29723ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [29726ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [30141ms] querying master, [30143ms] Sub > > rpc: ConnectToMaster sending RPC to server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, > > [30144ms] Sub rpc: ConnectToMaster received from server > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > response Network error: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [30147ms] delaying RPC due to Service > > unavailable: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed, [33361ms] trace too long, truncated) > > CAUSED BY: NoLeaderFoundException: Master config > > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no > > leader. Exceptions received: > > org.apache.kudu.client.RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed > > CAUSED BY: RecoverableException: [Peer > > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] > > Connection closed > > > > I also added to Impala Command Line Argument Advanced > > Configuration Snippet (Safety Valve) the setting: > > -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051> > > > > And to Impala Service Environment Advanced Configuration > > Snippet (Safety Valve) the setting: > > IMPALA_KUDU=1 > > > > I don't think these are correct because it's not working. > > > > Cheers, > > Ben > > > > > > On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com > > <mailto:t...@cloudera.com>> wrote: > > > > Hi Ben, > > > > What error are you getting? The Impala shell shouldn't be > > relevant here -- it's just submitting queries to the > > Impala daemons themselves. > > > > This deployment scenario hasn't been tested much as far as > > I know, so I wouldn't be surprised if there are issues > > with scheduling fragments given lack of locality, etc, but > > I would expect it to at least "basically work". (I've done > > it once or twice to copy data from one cluster to another) > > > > -Todd > > > > On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim > > <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: > > > > Can someone help me with configuring Impala using > > Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I > > cannot get it to connect using impala shell. > > > > Cheers, > > Ben > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > > >
