[
https://issues.apache.org/jira/browse/KUDU-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2830:
---------------------------------
Assignee: Grant Henke
> Improve RPC traces on RPC timeout
> ---------------------------------
>
> Key: KUDU-2830
> URL: https://issues.apache.org/jira/browse/KUDU-2830
> Project: Kudu
> Issue Type: Improvement
> Reporter: Grant Henke
> Assignee: Grant Henke
> Priority: Major
> Labels: backup
>
> When RPCs are retried repeatedly and then eventually a timeout is caused, the
> logged RPC trace includes the entire nested RPC call stack. This is often way
> to much to read and usually includes a lot of repeated information. Perhaps
> we could use various levels of detail and summarization based on the log
> level to make things more useful.
> Below is a sample RPC trace:
> {noformat}
> 19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage
> 1.0 (TID 1152, vc1308.halxg.cloudera.com, executor 9):
> java.lang.RuntimeException: failed to write 516 rows from DataFrame to Kudu;
> sample errors: Timed out: can not complete before timeout:
> Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
> [0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
> rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a,
> attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms]
> querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
