[
https://issues.apache.org/jira/browse/OOZIE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426558#comment-17426558
]
Dénes Bodó commented on OOZIE-3640:
-----------------------------------
Hey [~puru]
Thank you for the investigation and suggestion. I suppose you suggest this as a
file system action. Do you have any proposal code for it? Or some performance
measurements?
I think if there is a need to check any HDFS directory size then a simple Java
application run as a Java action could do the trick and that does not block
Oozie. Anyway, if this dirSize fs action works then please share your change
with tests and some performance measurements.
Thanks,
Denes
> Add support for recursive directories fs:dirSize
> ------------------------------------------------
>
> Key: OOZIE-3640
> URL: https://issues.apache.org/jira/browse/OOZIE-3640
> Project: Oozie
> Issue Type: Improvement
> Reporter: Purshotam Shah
> Priority: Major
>
> There are three ways to do that.
> # Use getContentSummary: This can be dangerous for name nodes.
> getContentSummary on a large HDFS dirs can take minutes. During that time, an
> oozie thread will be blocked, waiting for the RPC response. A 5 min workflow
> doing a content summary on a directory that takes > 5 min may lead to oozie
> thread exhaustion. or if oozie times out and retries, the NN has no support
> for aborting a call being processed, so now there will be multiple concurrent
> content summaries which may also exhaust the NN's handler threads.
> # Use getQuotaUsage: If quote is not enabled, it will fall back on
> getContentSummary. So this is as bad as getContentSummary.
> # Use recursive listing to compute size. Enforce system-level dir size(or
> file count) and recursive level if it reached the max-level or max-size throw
> exception.
> Considering all three options.
> Option 3 is the best option. A system admin can configure max-level based on
> system load and user use-cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
