[
https://issues.apache.org/jira/browse/HADOOP-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218072#comment-15218072
]
Steve Loughran commented on HADOOP-12979:
-----------------------------------------
The problem is the branch take if there's no s3 buffer dir defined. the else
clause is broken; it's looking for a config option which isn't there.
{code}
if (conf.get(BUFFER_DIR, null) != null) {
lDirAlloc = new LocalDirAllocator(BUFFER_DIR);
} else {
lDirAlloc = new LocalDirAllocator("${hadoop.tmp.dir}/s3a"); // HERE
}
{code}
The fix should be to set {{BUFFER_DIR}} to the full path desired, create the
{{LocalDirAllocator(BUFFER_DIR)}} from the (possibly enhanced) config
> IOE in S3a: ${hadoop.tmp.dir}/s3a not configured
> -------------------------------------------------
>
> Key: HADOOP-12979
> URL: https://issues.apache.org/jira/browse/HADOOP-12979
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
>
> Running some spark s3a tests trigger an NPE in Hadoop <=2/7l IOE in 2.8
> saying "${hadoop.tmp.dir}/s3a not configured".
> That's correct: there is no configuration option on the conf called
> "${hadoop.tmp.dir}/s3a ". There may be one called {{hadoop.tmp.dir}}, however.
> Essentially s3a is sending the wrong config option down, if it can't find
> {{fs.s3a.buffer.dir}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
