[ 
https://issues.apache.org/jira/browse/KUDU-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke resolved KUDU-2830.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.10.0

Resolved via 
[https://github.com/apache/kudu/commit/4fe36192412bd74dd3b3518bff1f9a36a8d60155]

> Improve RPC traces on RPC timeout
> ---------------------------------
>
>                 Key: KUDU-2830
>                 URL: https://issues.apache.org/jira/browse/KUDU-2830
>             Project: Kudu
>          Issue Type: Improvement
>            Reporter: Grant Henke
>            Assignee: Grant Henke
>            Priority: Major
>              Labels: backup
>             Fix For: 1.10.0
>
>
> When RPCs are retried repeatedly and then eventually a timeout is caused, the 
> logged RPC trace includes the entire nested RPC call stack. This is often way 
> to much to read and usually includes a lot of repeated information. Perhaps 
> we could use various levels of detail and summarization based on the log 
> level to make things more useful. 
> Below is a sample RPC trace:
> {noformat}
> 19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage 
> 1.0 (TID 1152, vc1308.halxg.cloudera.com, executor 9): 
> java.lang.RuntimeException: failed to write 516 rows from DataFrame to Kudu; 
> sample errors: Timed out: can not complete before timeout: 
> Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a" 
> [0x0000000300000003, <end>), ignoreAllDuplicateRows=false, 
> rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, 
> attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] 
> querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from 
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] 
> delaying RPC due to Service unavailable: Service unavailable: Soft memory 
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub 
> rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [97ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] 
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [160ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] 
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [262ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] 
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [516ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] 
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [758ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] 
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [916ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] 
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of 
> capacity), [1103ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] 
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of 
> capacity), [4524ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] 
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of 
> capacity), [7061ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] 
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of 
> capacity), [9860ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received 
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: 
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), 
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft 
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, 
> [16920ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of 
> capacity), [16957ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] 
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of 
> capacity), [19002ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] 
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of 
> capacity), [22476ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] 
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of 
> capacity), [24562ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] 
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of 
> capacity), [25117ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] 
> querying master, [28740ms] trace too long, truncated)}Timed out: can not 
> complete before timeout: Batch{operations=516, 
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), 
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, 
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, 
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, 
> [0ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from 
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] 
> delaying RPC due to Service unavailable: Service unavailable: Soft memory 
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub 
> rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [97ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] 
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [160ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] 
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [262ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] 
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [516ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] 
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [758ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] 
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [916ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] 
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of 
> capacity), [1103ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] 
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of 
> capacity), [4524ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] 
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of 
> capacity), [7061ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] 
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of 
> capacity), [9860ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received 
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: 
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), 
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft 
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, 
> [16920ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of 
> capacity), [16957ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] 
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of 
> capacity), [19002ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] 
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of 
> capacity), [22476ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] 
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of 
> capacity), [24562ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] 
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of 
> capacity), [25117ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] 
> querying master, [28740ms] trace too long, truncated)}Timed out: can not 
> complete before timeout: Batch{operations=516, 
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), 
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, 
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, 
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, 
> [0ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from 
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] 
> delaying RPC due to Service unavailable: Service unavailable: Soft memory 
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub 
> rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [97ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] 
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [160ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] 
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [262ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] 
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [516ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] 
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [758ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] 
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [916ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] 
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of 
> capacity), [1103ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] 
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of 
> capacity), [4524ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] 
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of 
> capacity), [7061ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] 
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of 
> capacity), [9860ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received 
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: 
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), 
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft 
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, 
> [16920ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of 
> capacity), [16957ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] 
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of 
> capacity), [19002ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] 
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of 
> capacity), [22476ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] 
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of 
> capacity), [24562ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] 
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of 
> capacity), [25117ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] 
> querying master, [28740ms] trace too long, truncated)}Timed out: can not 
> complete before timeout: Batch{operations=516, 
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), 
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, 
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, 
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, 
> [0ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from 
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] 
> delaying RPC due to Service unavailable: Service unavailable: Soft memory 
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub 
> rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [97ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] 
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [160ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] 
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [262ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] 
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [516ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] 
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [758ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] 
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [916ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] 
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of 
> capacity), [1103ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] 
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of 
> capacity), [4524ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] 
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of 
> capacity), [7061ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] 
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of 
> capacity), [9860ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received 
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: 
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), 
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft 
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, 
> [16920ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of 
> capacity), [16957ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] 
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of 
> capacity), [19002ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] 
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of 
> capacity), [22476ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] 
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of 
> capacity), [24562ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] 
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of 
> capacity), [25117ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] 
> querying master, [28740ms] trace too long, truncated)}Timed out: can not 
> complete before timeout: Batch{operations=516, 
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>), 
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, 
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35, 
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master, 
> [0ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from 
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms] 
> delaying RPC due to Service unavailable: Service unavailable: Soft memory 
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub 
> rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [97ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms] 
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [160ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms] 
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [262ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms] 
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [516ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms] 
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [758ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms] 
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of 
> capacity), [916ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms] 
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of 
> capacity), [1103ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms] 
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of 
> capacity), [4524ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms] 
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of 
> capacity), [7061ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms] 
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of 
> capacity), [9860ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms] 
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received 
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: 
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity), 
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft 
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master, 
> [16920ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of 
> capacity), [16957ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms] 
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of 
> capacity), [19002ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms] 
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of 
> capacity), [22476ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms] 
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of 
> capacity), [24562ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms] 
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server 
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations 
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, 
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] 
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service 
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of 
> capacity), [25117ms] delaying RPC due to Service unavailable: Service 
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms] 
> querying master, [28740ms] trace too long, truncated)}
> at 
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)
> at 
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)
> at 
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at 
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to