[
https://issues.apache.org/jira/browse/MAPREDUCE-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7317.
---------------------------------------
Fix Version/s: 3.3.1
Resolution: Fixed
Fixed in branch-3.3+
> Add latency information in FileOutputCommitter.mergePaths
> ---------------------------------------------------------
>
> Key: MAPREDUCE-7317
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7317
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We have been observed some occurrences of huge delay from file output
> committer V1, where file output committer V2 is not an option.
> While the root cause should have investigated on our side, there's another
> issue that there's insufficient information to debug. Most likely the huge
> delay comes from mergePaths, but the class only provides the "debug" log
> message to log the call itself with parameters, nothing else. mergePaths has
> been called recursively which is harder to trace how much latency specific
> directory takes to merge.
> It would be nice and not intrusive to add latency info in mergePath, so that
> we can see how much latency specific directory takes to merge, only when
> debug log is enabled.
> (Ideally it'd be nice if we can log warn message when the call takes huge
> time to process, but I don't have the proper threshold for the "huge time",
> so I'd avoid dealing with it altogether here.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
