[
https://issues.apache.org/jira/browse/KUDU-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854956#comment-16854956
]
Grant Henke edited comment on KUDU-2830 at 6/3/19 7:07 PM:
-----------------------------------------------------------
A proposed "trace summary" format would be:
{noformat}
Trace Summary(ms): Sent(n), Received(n), Delayed(n), MasterRefresh(n),
AuthWait(n), Truncated: ?
Sent: (uuid, [ method, count ], ...), ...
Received: (uuid, [ status, count ], ...), ...
Delayed: (uuid, [ method, count ], ...), ...
{noformat}
The summary style would be printed for _INFO_ level logging or above, and the
full trace for _DEBUG_ level or below.
Based on one of the traces from the above example this would look like:
{noformat}
Trace Summary(28740ms): Sent(34), Received(34), Delayed(17), MasterRefresh(17),
AuthWait(0), Truncated: true
Sent: (c31055f8d3f74ee3b52687e7aff5b68a, [ GetTableLocations, 17 ]),
(09d6bf7a02124145b43f43cb7a667b3d, [ Batch, 17])
Received: (c31055f8d3f74ee3b52687e7aff5b68a, [ OK, 17 ]),
(09d6bf7a02124145b43f43cb7a667b3d, [ SERVICE_UNAVAILABLE, 17])
Delayed: (09d6bf7a02124145b43f43cb7a667b3d, [ Batch, 17 ]){noformat}
We could also specially handle _SERVICE_UNAVAILABLE_ when it's from a soft
memory limit rejection given that is fairly common and distinct from other
_SERVICE_UNAVAILABLE_ errors.
was (Author: granthenke):
A proposed "trace summary" format would be:
{noformat}
Trace Summary(ms): Sent(n), Received(n), Delayed(n), MasterRefresh(n),
AuthWait(n), Truncated: ?
Sent: (uuid, [ method, count ], ...), ...
Received: (uuid, [ status, count ], ...), ...
Delayed: (uuid, [ method, count ], ...), ...
{noformat}
The summary style would be printed for INFO level logging or above, and the
full trace for DEBUG level or below.
Based on one of the traces from the above example this would look like:
{noformat}
Trace Summary(28740ms): Sent(34), Received(34), Delayed(17), MasterRefresh(17),
AuthWait(0), Truncated: true
Sent: (c31055f8d3f74ee3b52687e7aff5b68a, [ GetTableLocations, 17 ]),
(09d6bf7a02124145b43f43cb7a667b3d, [ Batch, 17])
Received: (c31055f8d3f74ee3b52687e7aff5b68a, [ OK, 17 ]),
(09d6bf7a02124145b43f43cb7a667b3d, [ SERVICE_UNAVAILABLE, 17])
Delayed: (09d6bf7a02124145b43f43cb7a667b3d, [ Batch, 17 ]){noformat}
We could also specially handle `SERVICE_UNAVAILABLE` when it's from a soft
memory limit rejection given that is fairly common and distinct from other
`SERVICE_UNAVAILABLE` errors.
> Improve RPC traces on RPC timeout
> ---------------------------------
>
> Key: KUDU-2830
> URL: https://issues.apache.org/jira/browse/KUDU-2830
> Project: Kudu
> Issue Type: Improvement
> Reporter: Grant Henke
> Assignee: Grant Henke
> Priority: Major
> Labels: backup
>
> When RPCs are retried repeatedly and then eventually a timeout is caused, the
> logged RPC trace includes the entire nested RPC call stack. This is often way
> to much to read and usually includes a lot of repeated information. Perhaps
> we could use various levels of detail and summarization based on the log
> level to make things more useful.
> Below is a sample RPC trace:
> {noformat}
> 19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage
> 1.0 (TID 1152, vc1308.halxg.cloudera.com, executor 9):
> java.lang.RuntimeException: failed to write 516 rows from DataFrame to Kudu;
> sample errors: Timed out: can not complete before timeout:
> Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
> [0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
> rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a,
> attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms]
> querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
