Todd, I'll check this with the systems team.
Cheers, Ben On Mon, Aug 14, 2017 at 10:40 PM Todd Lipcon <t...@cloudera.com> wrote: > Is there a possibility that the remote node (prod-dc1-datanode151) is > firewalled off from whatever host you are submitting the query to? The > error message is admittedly pretty bad, but it basically means it's getting > "connection refused", indicating that either there is no master running on > that host or it has been blocked (eg an iptables REJECT rule) > > -Todd > > On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> Hi Todd, >> >> I tried to create a Kudu table using impala shell, and I got this error. >> >> create table my_first_table >> ( >> id bigint, >> name string, >> primary key(id) >> ) >> partition by hash partitions 16 >> stored as kudu; >> Query: create table my_first_table >> ( >> id bigint, >> name string, >> primary key(id) >> ) >> partition by hash partitions 16 >> stored as kudu >> ERROR: ImpalaRuntimeException: Error creating Kudu table >> 'impala::default.my_first_table' >> CAUSED BY: NonRecoverableException: Too many attempts: >> KuduRpc(method=ListTables, tablet=null, attempt=101, >> DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] querying >> master, [1ms] Sub rpc: ConnectToMaster sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [5ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [21ms] querying master, [22ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [24ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [26ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [41ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [43ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [46ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [62ms] querying master, [62ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [64ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [66ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [81ms] querying master, [81ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [84ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [86ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [122ms] querying master, [122ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [123ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [124ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [161ms] querying master, [162ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [163ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [166ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [302ms] querying master, [302ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [304ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [306ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [382ms] querying master, [382ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [384ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [386ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [622ms] querying master, [622ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [623ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [626ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [701ms] querying master, [702ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [703ms] Sub >> rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [706ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [2062ms] querying master, [2062ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [2065ms] >> Sub rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [2068ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [4661ms] querying master, [4662ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [4663ms] >> Sub rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [4666ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [6961ms] querying master, [6962ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [6964ms] >> Sub rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [6967ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [9102ms] querying master, [9102ms] Sub rpc: ConnectToMaster sending RPC to >> server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [9104ms] >> Sub rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [9106ms] delaying RPC due to Service unavailable: Master >> config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. >> Exceptions received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed, >> [12001ms] querying master, [12002ms] Sub rpc: ConnectToMaster sending RPC >> to server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, >> [12003ms] Sub rpc: ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [12006ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [15822ms] querying master, [15822ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [15824ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [15826ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [16002ms] querying master, [16002ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [16004ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [16006ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [18381ms] querying master, [18381ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18383ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [18386ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [18821ms] querying master, [18822ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [18823ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [18826ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [19262ms] querying master, [19262ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [19263ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [19267ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [23082ms] querying master, [23082ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [23083ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [23086ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [26062ms] querying master, [26063ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [26064ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [26067ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [29721ms] querying master, [29722ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [29723ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [29726ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [30141ms] querying master, [30143ms] Sub rpc: ConnectToMaster >> sending RPC to server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051, [30144ms] Sub rpc: >> ConnectToMaster received from server >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 response Network >> error: [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] >> Connection closed, [30147ms] delaying RPC due to Service unavailable: >> Master config (prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no >> leader. Exceptions received: org.apache.kudu.client.RecoverableException: >> [Peer master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection >> closed, [33361ms] trace too long, truncated) >> CAUSED BY: NoLeaderFoundException: Master config ( >> prod-dc1-datanode151.pdc1i.gradientx.com:7051) has no leader. Exceptions >> received: org.apache.kudu.client.RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed >> CAUSED BY: RecoverableException: [Peer >> master-prod-dc1-datanode151.pdc1i.gradientx.com:7051] Connection closed >> >> I also added to Impala Command Line Argument Advanced Configuration >> Snippet (Safety Valve) the setting: >> -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> >> And to Impala Service Environment Advanced Configuration Snippet (Safety >> Valve) the setting: >> IMPALA_KUDU=1 >> >> I don't think these are correct because it's not working. >> >> Cheers, >> Ben >> >> >> On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com> wrote: >> >>> Hi Ben, >>> >>> What error are you getting? The Impala shell shouldn't be relevant here >>> -- it's just submitting queries to the Impala daemons themselves. >>> >>> This deployment scenario hasn't been tested much as far as I know, so I >>> wouldn't be surprised if there are issues with scheduling fragments given >>> lack of locality, etc, but I would expect it to at least "basically work". >>> (I've done it once or twice to copy data from one cluster to another) >>> >>> -Todd >>> >>> On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim <bbuil...@gmail.com> >>> wrote: >>> >>>> Can someone help me with configuring Impala using Cloudera Manager for >>>> Kudu 1.4.0 on CDH 5.12.0? I cannot get it to connect using impala shell. >>>> >>>> Cheers, >>>> Ben >>>> >>>> >>> >>> >>> -- >>> Todd Lipcon >>> Software Engineer, Cloudera >>> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera >
