Is there a possibility that the remote node (prod-dc1-datanode151) is firewalled off from whatever host you are submitting the query to? The error message is admittedly pretty bad, but it basically means it's getting "connection refused", indicating that either there is no master running on that host or it has been blocked (eg an iptables REJECT rule)
-Todd On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > Hi Todd, > > I tried to create a Kudu table using impala shell, and I got this error. > > create table my_first_table > ( > id bigint, > name string, > primary key(id) > ) > partition by hash partitions 16 > stored as kudu; > Query: create table my_first_table > ( > id bigint, > name string, > primary key(id) > ) > partition by hash partitions 16 > stored as kudu > ERROR: ImpalaRuntimeException: Error creating Kudu table > 'impala::default.my_first_table' > CAUSED BY: NonRecoverableException: Too many attempts: > KuduRpc(method=ListTables, tablet=null, attempt=101, > DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] querying > master, [1ms] Sub rpc: ConnectToMaster sending RPC to server > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2ms] Sub rpc: > ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [5ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [21ms] querying master, [22ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [24ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [26ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [41ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [43ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [46ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [62ms] querying master, [62ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [64ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [66ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [81ms] querying master, [81ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [84ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [86ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [122ms] querying master, [122ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [123ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [124ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [161ms] querying master, [162ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [163ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [166ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [302ms] querying master, [302ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [304ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [306ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [382ms] querying master, [382ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [384ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [386ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [622ms] querying master, [622ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [623ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [626ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [701ms] querying master, [702ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [703ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [706ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [2062ms] querying master, [2062ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2065ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [2068ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [4661ms] querying master, [4662ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [4663ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [4666ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [6961ms] querying master, [6962ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [6964ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [6967ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [9102ms] querying master, [9102ms] Sub rpc: ConnectToMaster sending RPC to > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [9104ms] Sub > rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [9106ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [12001ms] querying master, [12002ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [12003ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [12006ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [15822ms] querying master, [15822ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [15824ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [15826ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [16002ms] querying master, [16002ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [16004ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [16006ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [18381ms] querying master, [18381ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18383ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [18386ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [18821ms] querying master, [18822ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18823ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [18826ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [19262ms] querying master, [19262ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [19263ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [19267ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [23082ms] querying master, [23082ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [23083ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [23086ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [26062ms] querying master, [26063ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [26064ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [26067ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [29721ms] querying master, [29722ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [29723ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [29726ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [30141ms] querying master, [30143ms] Sub rpc: ConnectToMaster sending RPC > to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [30144ms] > Sub rpc: ConnectToMaster received from server master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051 response Network error: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [30147ms] delaying RPC due to Service unavailable: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, > [33361ms] trace too long, truncated) > CAUSED BY: NoLeaderFoundException: Master config ( > prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions > received: org.apache.kudu.client.RecoverableException: [Peer > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed > CAUSED BY: RecoverableException: [Peer master-prod-dc1-datanode151. > pdc1i.gradientx.com:7051] Connection closed > > I also added to Impala Command Line Argument Advanced Configuration > Snippet (Safety Valve) the setting: > -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051 > > And to Impala Service Environment Advanced Configuration Snippet (Safety > Valve) the setting: > IMPALA_KUDU=1 > > I don't think these are correct because it's not working. > > Cheers, > Ben > > > On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com> wrote: > >> Hi Ben, >> >> What error are you getting? The Impala shell shouldn't be relevant here >> -- it's just submitting queries to the Impala daemons themselves. >> >> This deployment scenario hasn't been tested much as far as I know, so I >> wouldn't be surprised if there are issues with scheduling fragments given >> lack of locality, etc, but I would expect it to at least "basically work". >> (I've done it once or twice to copy data from one cluster to another) >> >> -Todd >> >> On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >> >>> Can someone help me with configuring Impala using Cloudera Manager for >>> Kudu 1.4.0 on CDH 5.12.0? I cannot get it to connect using impala shell. >>> >>> Cheers, >>> Ben >>> >>> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > -- Todd Lipcon Software Engineer, Cloudera
