[
https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-90:
------------------------------
Attachment: apache-yarn-90.7.patch
Uploaded a new patch to address the comments by Jason.
{quote}
bq. I've changed it to "Disk(s) health report: ". My only concern with this
is that there might be scripts looking for the "Disk(s) failed" log line for
monitoring. What do you think?
If that's true then the code should bother to do a diff between the old disk
list and the new one, logging which disks turned bad using the "Disk(s) failed"
line and which disks became healthy with some other log message.
{quote}
Fixed. We now have two log messages - one indicating when disks go bad and one
when disks get marked as good.
{quote}
bq. Directories are only cleaned up during startup. The code tests for
existence of the directories and the correct permissions. This does mean that
container directories left behind for any reason won't get cleaned up unit the
NodeManager is restarted. Is that ok?
This could still be problematic for the NM work-preserving restart case, as we
could try to delete an entire disk tree with active containers on it due to a
hiccup when the NM restarts. I think a better approach is a periodic cleanup
scan that looks for directories under yarn-local and yarn-logs that shouldn't
be there. This could be part of the health check scan or done separately. That
way we don't have to wait for a disk to turn good or bad to catch leaked
entities on the disk due to some hiccup. Sorta like an fsck for the NM state on
disk. That is best done as a separate JIRA, as I think this functionality is
still an incremental improvement without it.
{quote}
The current code will only cleanup if the NM recovery can't be carried out.
{noformat}
if (!stateStore.canRecover()) {
cleanUpLocalDirs(lfs, delService);
initializeLocalDirs(lfs);
initializeLogDirs(lfs);
}
{noformat}
Will that handle the case you mentioned?
bq. checkDirs unnecessarily calls union(errorDirs, fullDirs) twice.
Fixed.
bq. isDiskFreeSpaceOverLimt is now named backwards, as the code returns true if
the free space is under the limit.
Fixed.
bq. getLocalDirsForCleanup and getLogDirsForCleanup should have javadoc
comments like the other methods.
Fixed.
{quote}
Nit: The union utility function doesn't technically perform a union but rather
a concatenation, and it'd be a little clearer if the name reflected that. Also
the function should leverage the fact that it knows how big the ArrayList will
be after the operations and give it the appropriate hint to its constructor to
avoid reallocations.
{quote}
Fixed.
> NodeManager should identify failed disks becoming good back again
> -----------------------------------------------------------------
>
> Key: YARN-90
> URL: https://issues.apache.org/jira/browse/YARN-90
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Reporter: Ravi Gummadi
> Assignee: Varun Vasudev
> Attachments: YARN-90.1.patch, YARN-90.patch, YARN-90.patch,
> YARN-90.patch, YARN-90.patch, apache-yarn-90.0.patch, apache-yarn-90.1.patch,
> apache-yarn-90.2.patch, apache-yarn-90.3.patch, apache-yarn-90.4.patch,
> apache-yarn-90.5.patch, apache-yarn-90.6.patch, apache-yarn-90.7.patch
>
>
> MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes
> down, it is marked as failed forever. To reuse that disk (after it becomes
> good), NodeManager needs restart. This JIRA is to improve NodeManager to
> reuse good disks(which could be bad some time back).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
