On Wed, Aug 16, 2017 at 6:16 AM, Benjamin Kim <bbuil...@gmail.com> wrote:
> Hi, > > I found 2 issues. First, network connection is blocked. I filed a request > to open it. Second, Kudu 1.4 had problems registering any tablet server. > So, I reverted back to 1.3.1. The error had to do security and how IP > routing was not trusted. > Can you clarify what problem you are hitting? Is this the security improvement that prevents from running with public IP addresses without security enabled? You can override this if you're sure that your network is secured by using the --trusted-subnets option (see the security docs for details) -Todd > > Also, I installed Kudu onto another cluster where I knew the network would > not be a problem, and it worked. > > Cheers, > Ben > > On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin <aser...@cloudera.com> > wrote: > >> Ben, >> >> As Todd mentioned, it might be some network connectivity problem. I >> would suspect some issues with connectivity between the node where the >> Impala shell is running and the Kudu master node. >> >> To start troubleshooting, I would verify that the node where you run the >> Impala shell (that's 172.35.120.191, right?) can establish a TCP >> connection to the master RPC end-point. E.g., try to run from the >> command shell at 172.35.120.191: >> >> telnet 172.35.121.101 7051 >> >> Would it succeed? >> >> Also, if running multi-master Kudu cluster, it might happen that masters >> cannot communicate with each other. To troubleshoot that, I would try >> to establish a TCP connection to the RPC end-point of the master at one >> node from another master node. E.g., if using telnet, from >> <master_node_0>, in the command-line shell: >> >> telnet <master_node_1> 7051 >> (just substitute <master_node_0> and <master_node_1> with appropriate >> hostnames/IP addresses). >> >> >> >> Best regards, >> >> Alexey >> >> >> On 8/15/17 9:53 PM, Benjamin Kim wrote: >> > Todd, >> > >> > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master >> > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer Kudu Master - >> > prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection >> > reset on [id: 0x6232f33f, /172.35.120.191:47848 >> > <http://172.35.120.191:47848/> :> /172.35.121.101:7051 >> > <http://172.35.121.101:7051/>] >> > >> > We got this error trying to use Kudu from within the cluster. Do you >> > know what this means? >> > >> > Cheers, >> > Ben >> > >> > >> > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <t...@cloudera.com >> > <mailto:t...@cloudera.com>> wrote: >> > >> > Is there a possibility that the remote node (prod-dc1-datanode151) >> > is firewalled off from whatever host you are submitting the query >> > to? The error message is admittedly pretty bad, but it basically >> > means it's getting "connection refused", indicating that either >> > there is no master running on that host or it has been blocked (eg >> > an iptables REJECT rule) >> > >> > -Todd >> > >> > On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com >> > <mailto:bbuil...@gmail.com>> wrote: >> > >> > Hi Todd, >> > >> > I tried to create a Kudu table using impala shell, and I got >> > this error. >> > >> > create table my_first_table >> > ( >> > id bigint, >> > name string, >> > primary key(id) >> > ) >> > partition by hash partitions 16 >> > stored as kudu; >> > Query: create table my_first_table >> > ( >> > id bigint, >> > name string, >> > primary key(id) >> > ) >> > partition by hash partitions 16 >> > stored as kudu >> > ERROR: ImpalaRuntimeException: Error creating Kudu table >> > 'impala::default.my_first_table' >> > CAUSED BY: NonRecoverableException: Too many attempts: >> > KuduRpc(method=ListTables, tablet=null, attempt=101, >> > DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] >> > querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to >> > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [2ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [5ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [21ms] querying master, [22ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [24ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [26ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [41ms] querying master, [41ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [43ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [46ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [62ms] querying master, [62ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [64ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [66ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [81ms] querying master, [81ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [84ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [86ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [122ms] querying master, [122ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [123ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [124ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [161ms] querying master, [162ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [163ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [166ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [302ms] querying master, [302ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [304ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [306ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [382ms] querying master, [382ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [384ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [386ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [622ms] querying master, [622ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [623ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [626ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [701ms] querying master, [702ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [703ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [706ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [2062ms] querying master, [2062ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [2065ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [2068ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [4661ms] querying master, [4662ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [4663ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [4666ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [6961ms] querying master, [6962ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [6964ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [6967ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [9102ms] querying master, [9102ms] Sub rpc: >> > ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [9104ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [9106ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [12001ms] querying master, [12002ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [12003ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [12006ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [15822ms] querying master, [15822ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [15824ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [15826ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [16002ms] querying master, [16002ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [16004ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [16006ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [18381ms] querying master, [18381ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [18383ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [18386ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [18821ms] querying master, [18822ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [18823ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [18826ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [19262ms] querying master, [19262ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [19263ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [19267ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [23082ms] querying master, [23082ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [23083ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [23086ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [26062ms] querying master, [26063ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [26064ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [26067ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [29721ms] querying master, [29722ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [29723ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [29726ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [30141ms] querying master, [30143ms] Sub >> > rpc: ConnectToMaster sending RPC to server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >> > [30144ms] Sub rpc: ConnectToMaster received from server >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > response Network error: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [30147ms] delaying RPC due to Service >> > unavailable: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed, [33361ms] trace too long, truncated) >> > CAUSED BY: NoLeaderFoundException: Master config >> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >> > leader. Exceptions received: >> > org.apache.kudu.client.RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed >> > CAUSED BY: RecoverableException: [Peer >> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >> > Connection closed >> > >> > I also added to Impala Command Line Argument Advanced >> > Configuration Snippet (Safety Valve) the setting: >> > -kudu_master_hosts=prod-dc1-datanode151.pdc1i.gradientx. >> com:7051 >> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051> >> > >> > And to Impala Service Environment Advanced Configuration >> > Snippet (Safety Valve) the setting: >> > IMPALA_KUDU=1 >> > >> > I don't think these are correct because it's not working. >> > >> > Cheers, >> > Ben >> > >> > >> > On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com >> > <mailto:t...@cloudera.com>> wrote: >> > >> > Hi Ben, >> > >> > What error are you getting? The Impala shell shouldn't be >> > relevant here -- it's just submitting queries to the >> > Impala daemons themselves. >> > >> > This deployment scenario hasn't been tested much as far as >> > I know, so I wouldn't be surprised if there are issues >> > with scheduling fragments given lack of locality, etc, but >> > I would expect it to at least "basically work". (I've done >> > it once or twice to copy data from one cluster to another) >> > >> > -Todd >> > >> > On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim >> > <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >> > >> > Can someone help me with configuring Impala using >> > Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I >> > cannot get it to connect using impala shell. >> > >> > Cheers, >> > Ben >> > >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> >> -- Todd Lipcon Software Engineer, Cloudera
