Todd, I'll keep this in mind. This information will be useful. I'll try again.
Thanks, Ben On Wed, Aug 16, 2017 at 4:32 PM Todd Lipcon <t...@cloudera.com> wrote: > On Wed, Aug 16, 2017 at 6:16 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> Hi, >> >> I found 2 issues. First, network connection is blocked. I filed a request >> to open it. Second, Kudu 1.4 had problems registering any tablet server. >> So, I reverted back to 1.3.1. The error had to do security and how IP >> routing was not trusted. >> > > Can you clarify what problem you are hitting? Is this the security > improvement that prevents from running with public IP addresses without > security enabled? You can override this if you're sure that your network is > secured by using the --trusted-subnets option (see the security docs for > details) > > -Todd > > >> >> Also, I installed Kudu onto another cluster where I knew the network >> would not be a problem, and it worked. >> >> Cheers, >> Ben >> >> On Wed, Aug 16, 2017 at 12:44 AM Alexey Serbin <aser...@cloudera.com> >> wrote: >> >>> Ben, >>> >>> As Todd mentioned, it might be some network connectivity problem. I >>> would suspect some issues with connectivity between the node where the >>> Impala shell is running and the Kudu master node. >>> >>> To start troubleshooting, I would verify that the node where you run the >>> Impala shell (that's 172.35.120.191, right?) can establish a TCP >>> connection to the master RPC end-point. E.g., try to run from the >>> command shell at 172.35.120.191: >>> >>> telnet 172.35.121.101 7051 >>> >>> Would it succeed? >>> >>> Also, if running multi-master Kudu cluster, it might happen that masters >>> cannot communicate with each other. To troubleshoot that, I would try >>> to establish a TCP connection to the RPC end-point of the master at one >>> node from another master node. E.g., if using telnet, from >>> <master_node_0>, in the command-line shell: >>> >>> telnet <master_node_1> 7051 >>> (just substitute <master_node_0> and <master_node_1> with appropriate >>> hostnames/IP addresses). >>> >>> >>> >>> Best regards, >>> >>> Alexey >>> >>> >>> On 8/15/17 9:53 PM, Benjamin Kim wrote: >>> > Todd, >>> > >>> > Caused by: org.apache.kudu.client.NoLeaderMasterFoundException: Master >>> > config (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer Kudu Master - >>> > prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051/>] Connection >>> > reset on [id: 0x6232f33f, /172.35.120.191:47848 >>> > <http://172.35.120.191:47848/> :> /172.35.121.101:7051 >>> > <http://172.35.121.101:7051/>] >>> > >>> > We got this error trying to use Kudu from within the cluster. Do you >>> > know what this means? >>> > >>> > Cheers, >>> > Ben >>> > >>> > >>> > On Tue, Aug 15, 2017 at 12:40 AM Todd Lipcon <t...@cloudera.com >>> > <mailto:t...@cloudera.com>> wrote: >>> > >>> > Is there a possibility that the remote node (prod-dc1-datanode151) >>> > is firewalled off from whatever host you are submitting the query >>> > to? The error message is admittedly pretty bad, but it basically >>> > means it's getting "connection refused", indicating that either >>> > there is no master running on that host or it has been blocked (eg >>> > an iptables REJECT rule) >>> > >>> > -Todd >>> > >>> > On Mon, Aug 14, 2017 at 10:36 PM, Benjamin Kim <bbuil...@gmail.com >>> > <mailto:bbuil...@gmail.com>> wrote: >>> > >>> > Hi Todd, >>> > >>> > I tried to create a Kudu table using impala shell, and I got >>> > this error. >>> > >>> > create table my_first_table >>> > ( >>> > id bigint, >>> > name string, >>> > primary key(id) >>> > ) >>> > partition by hash partitions 16 >>> > stored as kudu; >>> > Query: create table my_first_table >>> > ( >>> > id bigint, >>> > name string, >>> > primary key(id) >>> > ) >>> > partition by hash partitions 16 >>> > stored as kudu >>> > ERROR: ImpalaRuntimeException: Error creating Kudu table >>> > 'impala::default.my_first_table' >>> > CAUSED BY: NonRecoverableException: Too many attempts: >>> > KuduRpc(method=ListTables, tablet=null, attempt=101, >>> > DeadlineTracker(timeout=180000, elapsed=178226), Traces: [0ms] >>> > querying master, [1ms] Sub rpc: ConnectToMaster sending RPC to >>> > server master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [2ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [5ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [21ms] querying master, [22ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [24ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [26ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [41ms] querying master, [41ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [43ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [46ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [62ms] querying master, [62ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [64ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [66ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [81ms] querying master, [81ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [84ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [86ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [122ms] querying master, [122ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [123ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [124ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [161ms] querying master, [162ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [163ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [166ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [302ms] querying master, [302ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [304ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [306ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [382ms] querying master, [382ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [384ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [386ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [622ms] querying master, [622ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [623ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [626ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [701ms] querying master, [702ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [703ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [706ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [2062ms] querying master, [2062ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [2065ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [2068ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [4661ms] querying master, [4662ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [4663ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [4666ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [6961ms] querying master, [6962ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [6964ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [6967ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [9102ms] querying master, [9102ms] Sub rpc: >>> > ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [9104ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [9106ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [12001ms] querying master, [12002ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [12003ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [12006ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [15822ms] querying master, [15822ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [15824ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [15826ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [16002ms] querying master, [16002ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [16004ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [16006ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [18381ms] querying master, [18381ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [18383ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [18386ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [18821ms] querying master, [18822ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [18823ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [18826ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [19262ms] querying master, [19262ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [19263ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [19267ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [23082ms] querying master, [23082ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [23083ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [23086ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [26062ms] querying master, [26063ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [26064ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [26067ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [29721ms] querying master, [29722ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [29723ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [29726ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [30141ms] querying master, [30143ms] Sub >>> > rpc: ConnectToMaster sending RPC to server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>, >>> > [30144ms] Sub rpc: ConnectToMaster received from server >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > response Network error: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [30147ms] delaying RPC due to Service >>> > unavailable: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed, [33361ms] trace too long, truncated) >>> > CAUSED BY: NoLeaderFoundException: Master config >>> > (prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051>) has no >>> > leader. Exceptions received: >>> > org.apache.kudu.client.RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed >>> > CAUSED BY: RecoverableException: [Peer >>> > master-prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://master-prod-dc1-datanode151.pdc1i.gradientx.com:7051>] >>> > Connection closed >>> > >>> > I also added to Impala Command Line Argument Advanced >>> > Configuration Snippet (Safety Valve) the setting: >>> > -kudu_master_hosts= >>> prod-dc1-datanode151.pdc1i.gradientx.com:7051 >>> > <http://prod-dc1-datanode151.pdc1i.gradientx.com:7051> >>> > >>> > And to Impala Service Environment Advanced Configuration >>> > Snippet (Safety Valve) the setting: >>> > IMPALA_KUDU=1 >>> > >>> > I don't think these are correct because it's not working. >>> > >>> > Cheers, >>> > Ben >>> > >>> > >>> > On Mon, Aug 14, 2017 at 9:58 PM Todd Lipcon <t...@cloudera.com >>> > <mailto:t...@cloudera.com>> wrote: >>> > >>> > Hi Ben, >>> > >>> > What error are you getting? The Impala shell shouldn't be >>> > relevant here -- it's just submitting queries to the >>> > Impala daemons themselves. >>> > >>> > This deployment scenario hasn't been tested much as far as >>> > I know, so I wouldn't be surprised if there are issues >>> > with scheduling fragments given lack of locality, etc, but >>> > I would expect it to at least "basically work". (I've done >>> > it once or twice to copy data from one cluster to another) >>> > >>> > -Todd >>> > >>> > On Mon, Aug 14, 2017 at 7:59 PM, Benjamin Kim >>> > <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >>> > >>> > Can someone help me with configuring Impala using >>> > Cloudera Manager for Kudu 1.4.0 on CDH 5.12.0? I >>> > cannot get it to connect using impala shell. >>> > >>> > Cheers, >>> > Ben >>> > >>> > >>> > >>> > >>> > -- >>> > Todd Lipcon >>> > Software Engineer, Cloudera >>> > >>> > >>> > >>> > >>> > -- >>> > Todd Lipcon >>> > Software Engineer, Cloudera >>> > >>> >>> > > > -- > Todd Lipcon > Software Engineer, Cloudera >
