[jira] [Commented] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[
https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680365#comment-16680365
]
Steve Loughran commented on HADOOP-15915:
-
{code}
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid
local directory for s3ablock-0001-
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:447)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
at
org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
{code}
> Report problems w/ local S3A buffer directory meaningfully
> --
>
> Key: HADOOP-15915
> URL: https://issues.apache.org/jira/browse/HADOOP-15915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Steve Loughran
>Priority: Major
>
> When there's a problem working with the temp directory used for block output
> and the staging committers the actual path (and indeed config option) aren't
> printed.
> Improvements: tell the user which directory isn't writeable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[ https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680366#comment-16680366 ] Steve Loughran commented on HADOOP-15915: - +add to troubleshooting doc > Report problems w/ local S3A buffer directory meaningfully > -- > > Key: HADOOP-15915 > URL: https://issues.apache.org/jira/browse/HADOOP-15915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Priority: Major > > When there's a problem working with the temp directory used for block output > and the staging committers the actual path (and indeed config option) aren't > printed. > Improvements: tell the user which directory isn't writeable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[ https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082066#comment-17082066 ] Vasii Cosmin Radu commented on HADOOP-15915: I've encountered the same issue, Flink 1.10, EC2 deployment. Any ideas of how to fix this? The problem appears on Job Managers as soon as I submit a job. There is enough disk space and the Job Managers are not actually doing anything to need such much disk space, they are uploading job graph blobs. > Report problems w/ local S3A buffer directory meaningfully > -- > > Key: HADOOP-15915 > URL: https://issues.apache.org/jira/browse/HADOOP-15915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Priority: Major > > When there's a problem working with the temp directory used for block output > and the staging committers the actual path (and indeed config option) aren't > printed. > Improvements: tell the user which directory isn't writeable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[
https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621073#comment-17621073
]
Zbigniew Kostrzewa commented on HADOOP-15915:
-
I've recently stumbled upon this with {{{}3.2.2{}}}. For me the problem was
that I did not change {{hadoop.tmp.dir}} and so the {{s3ablock-0001-}} were
created in {{/tmp/hadoop-/s3a}} directory. At the same time,
on CentOS 7 in my case, there is a systemd service
{{systemd-tmpfiles-clean.service}} run once a day which cleans up {{/tmp}} of
files and directories older than 10 days. However, Node Manager after it caches
that {{/tmp/hadoop-/s3a}} exists it does not re-check it and
does not re-create that directory if it no longer exists, I believe the code
responsible for this is:
{code:java}
/** This method gets called everytime before any read/write to make sure
* that any change to localDirs is reflected immediately.
*/
private Context confChanged(Configuration conf)
throws IOException {
...
if (!newLocalDirs.equals(ctx.savedLocalDirs)) { {code}
and when the directory is missing log aggregation fails with this
{{DiskChecker}} error.
> Report problems w/ local S3A buffer directory meaningfully
> --
>
> Key: HADOOP-15915
> URL: https://issues.apache.org/jira/browse/HADOOP-15915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Steve Loughran
>Priority: Major
>
> When there's a problem working with the temp directory used for block output
> and the staging committers the actual path (and indeed config option) aren't
> printed.
> Improvements: tell the user which directory isn't writeable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[ https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079003#comment-17079003 ] Robert Metzger commented on HADOOP-15915: - +1 for improving the debugging experience here: https://lists.apache.org/thread.html/ref2db6f2367fbab60c8f54c6c3ae2c59ab82ddf602fe2046dac21822%40%3Cuser.flink.apache.org%3E > Report problems w/ local S3A buffer directory meaningfully > -- > > Key: HADOOP-15915 > URL: https://issues.apache.org/jira/browse/HADOOP-15915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Priority: Major > > When there's a problem working with the temp directory used for block output > and the staging committers the actual path (and indeed config option) aren't > printed. > Improvements: tell the user which directory isn't writeable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
