[
https://issues.apache.org/jira/browse/HADOOP-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300718#comment-15300718
]
Jason Lowe commented on HADOOP-10048:
-------------------------------------
bq. Given the ctx could be local context, I think we want to update it to the
currentContext which can accessed immediately - something like:
currentContext.get().dirNumLastAccessed = dirNum. Isn't it?
No, that could trigger an array bounds exception if we update it to a value
that is past the number of directories in the other, unrelated context. Also
we don't need to worry about this particular race. When the new context is set
it will compute a new random index for the next directory, so the index will
still be random even if a call using the old context missed updating the new
context.
bq. Shouldn't we check dir exists first then mkdir if not?
I was just preserving the existing behavior since it's unrelated to this
change. In practice the raw local fs mkdirs already does the exists check
anyway. Also switching them could lead to a weird error later if the local
path ended up being an existing file. localFS.exists will return true and we
won't try the mkdirs, but mkdirs throws a useful error message explaining it's
trying to create a directory where a file already exists. Since this is not
related to this change and could degrade the error diagnostics in some corner
cases, I'm tempted to leave it as-is. If we feel it's important to fix it then
we can tackle it in a followup JIRA where it does the full file stat first,
checks the corner cases, then calls mkdirs if necessary.
> LocalDirAllocator should avoid holding locks while accessing the filesystem
> ---------------------------------------------------------------------------
>
> Key: HADOOP-10048
> URL: https://issues.apache.org/jira/browse/HADOOP-10048
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 2.3.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: HADOOP-10048.003.patch, HADOOP-10048.004.patch,
> HADOOP-10048.005.patch, HADOOP-10048.patch, HADOOP-10048.trunk.patch
>
>
> As noted in MAPREDUCE-5584 and HADOOP-7016, LocalDirAllocator can be a
> bottleneck for multithreaded setups like the ShuffleHandler. We should
> consider moving to a lockless design or minimizing the critical sections to a
> very small amount of time that does not involve I/O operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
