[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130054#comment-14130054 ] Hudson commented on MAPREDUCE-6075: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1868 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1868/]) MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length files. Contributed by Jason Lowe (jlowe: rev 7f80e142096ab21b4852635a127589e34a747bc4) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130031#comment-14130031 ] Hudson commented on MAPREDUCE-6075: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1893 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1893/]) MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length files. Contributed by Jason Lowe (jlowe: rev 7f80e142096ab21b4852635a127589e34a747bc4) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java * hadoop-mapreduce-project/CHANGES.txt > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129909#comment-14129909 ] Hudson commented on MAPREDUCE-6075: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #677 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/677/]) MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length files. Contributed by Jason Lowe (jlowe: rev 7f80e142096ab21b4852635a127589e34a747bc4) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.6.0 > > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129219#comment-14129219 ] Jason Lowe commented on MAPREDUCE-6075: --- Thanks for the reviews, Tsuyoshi and Daryn! Committing this. > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127748#comment-14127748 ] Tsuyoshi OZAWA commented on MAPREDUCE-6075: --- [~daryn], thanks for your point, you're right. +1(non-binding) for Jason's change. http://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127432#comment-14127432 ] Daryn Sharp commented on MAPREDUCE-6075: I'm +1 on the change. The close/null/cleanup is a rather common pattern is hadoop. Using flush isn't a substitute for a close for all filesystems. Close must always be allowed to throw an exception and only swallowed when another exception occurred. In java, close() is supposed to be idempotent so double close is fine. Double closing a fd is bad because the fd may have already been recycled by another thread. > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123862#comment-14123862
]
Hadoop QA commented on MAPREDUCE-6075:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666918/MAPREDUCE-6075.patch
against trunk revision 21c0cde.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4857//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4857//console
This message is automatically generated.
> HistoryServerFileSystemStateStore can create zero-length files
> --
>
> Key: MAPREDUCE-6075
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.3.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: MAPREDUCE-6075.patch
>
>
> When the history server state store writes a token file it uses
> IOUtils.cleanup() to close the file which will silently ignore errors. This
> can lead to empty token files in the state store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123734#comment-14123734 ] Tsuyoshi OZAWA commented on MAPREDUCE-6075: --- [~jlowe], thanks for your contribution. IIUC, we shouldn't retry close after exception. Please check the following page for more detail: http://linux.die.net/man/2/close How about calling flush() instead of close() in try blocks? > HistoryServerFileSystemStateStore can create zero-length files > -- > > Key: MAPREDUCE-6075 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-6075.patch > > > When the history server state store writes a token file it uses > IOUtils.cleanup() to close the file which will silently ignore errors. This > can lead to empty token files in the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
